Nov 02 21:12:09.305238 ip-10-0-137-225 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:12:09.772398 ip-10-0-137-225 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:09.772398 ip-10-0-137-225 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:12:09.772398 ip-10-0-137-225 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:09.772398 ip-10-0-137-225 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:09.776576 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776498 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.776576 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776575 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.776576 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776579 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776582 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776585 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776588 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776590 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776593 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776596 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776599 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776602 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776606 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776610 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776613 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776616 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776618 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776621 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776624 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776627 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776629 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776632 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.776666 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776635 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776638 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776641 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776644 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776647 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776649 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776651 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776654 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776657 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776659 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776662 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776664 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776667 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776670 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776673 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776676 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776679 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776681 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776684 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776687 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.777119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776690 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776693 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776695 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776697 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776700 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776703 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776705 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776708 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776711 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776713 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776716 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776718 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776721 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776723 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776727 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776729 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776732 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776735 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776738 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776740 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.777624 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776743 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776745 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776748 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776750 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776753 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776755 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776759 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776762 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776764 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776767 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776769 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776772 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776776 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776781 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776784 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776786 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776789 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776792 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776794 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.778119 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776797 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776799 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776802 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776805 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776807 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776810 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776813 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776816 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776819 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776822 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776824 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776827 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776830 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776832 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776835 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776837 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776840 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776843 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776847 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776850 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.778590 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776852 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776854 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776857 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776860 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776862 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.776865 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777644 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777651 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777655 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777658 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777660 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777663 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777666 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777668 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777671 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777673 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777676 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777678 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777683 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.779079 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777686 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777688 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777691 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777694 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777697 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777699 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777701 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777704 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777707 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777709 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777712 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777714 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777717 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777719 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777722 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777724 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777727 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777730 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777733 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777736 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.779539 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777739 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777741 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777744 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777746 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777749 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777751 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777754 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777756 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777759 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777761 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777763 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777766 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777771 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777774 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777777 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777779 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777783 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777785 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777788 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777790 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.780018 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777793 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777796 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777799 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777801 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777804 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777806 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777809 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777811 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777814 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777816 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777819 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777821 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777825 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777827 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777829 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777832 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777834 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777837 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777841 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777846 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.780500 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777849 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777855 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777858 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777861 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777863 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777866 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777869 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777873 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777877 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777879 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777883 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777886 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777891 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777894 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777896 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777899 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777901 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.780982 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777904 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777906 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777909 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777911 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777914 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777917 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777920 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777922 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777925 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777928 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777930 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777933 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777936 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.777939 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778438 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778444 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778448 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778450 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778453 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.781556 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778456 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778459 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778461 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778465 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778469 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778472 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778475 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778478 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778481 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778483 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778486 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778489 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778491 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778494 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778496 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778499 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778502 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778504 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778510 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.782053 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778513 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778516 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778518 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778521 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778524 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778526 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778529 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778532 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778534 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778537 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778539 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778542 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778544 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778548 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778551 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778553 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778556 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778558 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778561 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.782502 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778563 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778566 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778568 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778570 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778573 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778575 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778578 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778580 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778583 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778585 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778588 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778590 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778595 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778598 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778601 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778604 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778608 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778610 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778613 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778615 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.782945 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778618 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778620 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778623 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778625 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778628 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778630 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778633 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778635 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778638 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778640 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778643 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778645 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778648 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778650 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778653 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778655 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778658 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778660 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778663 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778665 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.783428 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778668 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778670 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778673 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778675 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778679 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778681 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778684 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778687 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778690 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778692 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778695 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778697 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778700 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778702 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778705 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778707 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778710 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778712 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778715 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778717 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.783895 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778720 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778724 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.778727 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779199 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779213 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779219 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779224 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779229 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779232 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779237 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779240 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779243 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779247 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779251 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779254 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779257 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779261 2577 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779265 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779268 2577 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779270 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779276 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779281 2577 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779284 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:12:09.784359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779287 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779290 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779293 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779298 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779301 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779304 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779307 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779310 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779313 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779316 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779320 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779323 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779327 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779330 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779332 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779336 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779339 2577 flags.go:64] FLAG: --enable-server="true" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779342 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779346 2577 flags.go:64] FLAG: --event-burst="100" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779349 2577 flags.go:64] FLAG: --event-qps="50" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779352 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779355 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779359 2577 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779363 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:12:09.784882 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779366 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779385 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779390 2577 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779393 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779397 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779400 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779404 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779407 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779410 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779413 2577 flags.go:64] FLAG: --feature-gates="" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779417 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779421 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779424 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779427 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779430 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779433 2577 flags.go:64] FLAG: --help="false" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779436 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779439 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779443 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779446 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779449 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779453 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779456 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:12:09.785548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779458 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779461 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779464 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779467 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779470 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779473 2577 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779476 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779479 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779482 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779485 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779488 2577 flags.go:64] FLAG: --lock-file="" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779491 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779494 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779497 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779503 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779506 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779509 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779512 2577 flags.go:64] FLAG: --logging-format="text" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779515 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779518 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779521 2577 flags.go:64] FLAG: --manifest-url="" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779524 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779529 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779532 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779535 2577 flags.go:64] FLAG: --max-pods="110" Nov 02 21:12:09.786108 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779538 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779541 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779545 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779548 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779551 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779554 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779557 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779565 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779568 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779571 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779574 2577 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779577 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779583 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779586 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779589 2577 flags.go:64] FLAG: --port="10250" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779592 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779595 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-070cd3cbfd8637ccc" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779598 2577 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779601 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779604 2577 flags.go:64] FLAG: --register-node="true" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779608 2577 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779611 2577 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779615 2577 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779617 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779621 2577 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779625 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:12:09.786692 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779628 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779631 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779634 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779637 2577 flags.go:64] FLAG: --runonce="false" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779640 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779643 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779646 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779649 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779652 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779655 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779658 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779661 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779664 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779667 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779670 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779673 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779677 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779680 2577 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779684 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779689 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779692 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779695 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779702 2577 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779704 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779707 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:12:09.787328 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779710 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779713 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779716 2577 flags.go:64] FLAG: --v="2" Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779721 2577 flags.go:64] FLAG: --version="false" Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779725 2577 flags.go:64] FLAG: --vmodule="" Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779729 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.779733 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779866 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779869 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779872 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779875 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779878 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779880 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779883 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779885 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779888 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779890 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779894 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779898 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779901 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.787928 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779903 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779906 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779908 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779911 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779913 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779917 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779920 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779923 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779925 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779929 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779932 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779934 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779937 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779940 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779942 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779945 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779947 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779950 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779953 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.788425 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779956 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779959 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779961 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779976 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779979 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779982 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779985 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779987 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779991 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779993 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779996 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.779999 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780002 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780004 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780007 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780010 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780012 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780015 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780019 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780022 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.788878 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780024 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780027 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780031 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780034 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780037 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780040 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780043 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780046 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780048 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780051 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780053 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780056 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780059 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780062 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780064 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780067 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780069 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780072 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780074 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.789381 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780077 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780080 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780082 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780085 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780088 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780090 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780093 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780096 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780099 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780103 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780106 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780111 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780114 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780117 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780120 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780124 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780127 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780129 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780132 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.789833 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780135 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780138 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780141 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780143 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780146 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780149 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780151 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780154 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780157 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780159 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780162 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780164 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780167 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780170 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780172 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780175 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.780201 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:09.790375 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780235 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780240 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780243 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780245 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780248 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780251 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780253 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780256 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780260 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780263 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780266 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780270 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780272 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780275 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780278 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780281 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780283 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780286 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780289 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780291 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.790777 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780296 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780299 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780302 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780305 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780309 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780313 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780316 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780319 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780321 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780324 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780326 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780329 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780331 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780334 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780337 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780339 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780342 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780345 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.791251 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780347 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780352 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780355 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780358 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780361 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780364 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780367 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780369 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780372 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780375 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780377 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780380 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780382 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780385 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780388 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780390 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780393 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780396 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780399 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.791691 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780402 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780404 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780407 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780410 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780412 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780415 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780417 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780420 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780423 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780425 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780428 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780430 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780433 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780435 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780439 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780442 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780446 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780449 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780452 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780455 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.792147 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780457 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780459 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780462 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780464 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780467 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780470 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780472 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780475 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780477 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780480 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780482 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780485 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780488 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780491 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780493 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780496 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780500 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780504 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780508 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.792613 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780511 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780514 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780516 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780519 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780522 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780524 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780527 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780531 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.780534 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.787369 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.787384 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.788079 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.788085 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788171 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788176 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788179 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788183 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788185 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788188 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788191 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.793070 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788194 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788196 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788200 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788204 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788207 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788216 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788219 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788222 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788224 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788227 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788229 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788232 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788235 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788238 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788240 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788243 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788246 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788248 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788251 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.793557 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788253 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788256 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788259 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788262 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788265 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788267 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788270 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788273 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788275 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788278 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788281 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788283 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788286 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788288 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788291 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788296 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788299 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788301 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.794022 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788309 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788312 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788315 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788317 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788320 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788322 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788325 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788327 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788330 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788333 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788335 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788338 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788341 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788344 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788349 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788352 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788355 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788358 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788360 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788363 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.794465 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788366 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788368 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788371 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788373 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788376 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788378 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788381 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788383 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788386 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788388 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788391 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788394 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788396 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788404 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788407 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788409 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788412 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788414 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788417 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.794933 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788420 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788423 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788426 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788429 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788431 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788434 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788437 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788448 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788451 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788453 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788456 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788458 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788461 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788463 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788466 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788468 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788471 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788473 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788476 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788478 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.795427 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788480 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788483 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.788510 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788706 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788711 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788714 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788717 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788726 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788729 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788732 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788734 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788737 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788739 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788742 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788745 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788747 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.795884 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788750 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788752 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788755 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788757 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788760 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788762 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788765 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788767 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788770 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788772 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788774 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788777 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788780 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788782 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788784 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788787 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788790 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788792 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788795 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.796301 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788797 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788800 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788802 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788805 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788807 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788815 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788818 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788821 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788823 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788826 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788829 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788833 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788837 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788840 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788843 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788847 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788851 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788853 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788857 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.796754 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788859 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788862 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788864 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788867 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788869 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788872 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788874 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788877 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788879 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788882 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788885 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788888 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788890 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788893 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788896 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788899 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788901 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788903 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788906 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788914 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.797210 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788917 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788920 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788923 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788925 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788928 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788930 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788933 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788935 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788937 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788940 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788942 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788945 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788947 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788950 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788952 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788955 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788957 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788960 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788962 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788977 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.797889 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788980 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788983 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788986 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788989 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788992 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788994 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788997 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.788999 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789002 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789004 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789006 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789009 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789017 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789020 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789023 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.798645 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.789046 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789112 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789117 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789121 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789124 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789127 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789130 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789134 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789137 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789139 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789142 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789144 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789147 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789149 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789152 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789154 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789157 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789159 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789162 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:09.799302 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789165 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789167 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789170 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789172 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789175 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789177 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789180 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789183 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789185 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789188 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789190 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789201 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789204 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789206 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789209 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789211 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789214 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789216 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789219 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789221 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:09.800000 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789223 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789226 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789228 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789231 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789234 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789236 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789238 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789241 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789243 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789247 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789249 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789252 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789254 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789257 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789260 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789269 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789275 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789278 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789281 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:09.800698 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789284 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789286 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789289 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789292 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789294 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789303 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789307 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789310 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789312 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789315 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789317 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789320 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789323 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789325 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789327 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789330 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789333 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789335 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789338 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789340 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:09.801172 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789343 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789345 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789348 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789351 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789353 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789356 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789359 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789361 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789364 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789368 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789371 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789374 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789376 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789379 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789382 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789384 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789387 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789390 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789398 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:09.801674 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789401 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789403 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789406 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789409 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789411 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789421 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789423 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789426 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789429 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:09.789432 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.789481 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.792881 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.794234 2577 server.go:1014] "Starting client certificate rotation" Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.794363 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.794399 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.796058 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:09.802145 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.801225 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:09.811488 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.811473 2577 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:12:09.819712 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.819697 2577 log.go:25] "Validated CRI v1 image API" Nov 02 21:12:09.820468 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.820443 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:12:09.823679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.823657 2577 fs.go:135] Filesystem UUIDs: map[0d71475c-4762-40c2-a10d-1566d5a99b82:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a42eaacf-de30-455a-a963-001421f114e4:/dev/nvme0n1p3] Nov 02 21:12:09.823760 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.823677 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 21:12:09.828627 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.828518 2577 manager.go:217] Machine: {Timestamp:2026-11-02 21:12:09.827251887 +0000 UTC m=+0.394334120 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec226bba871b6f59b234e35cbc0302de SystemUUID:ec226bba-871b-6f59-b234-e35cbc0302de BootID:766251f0-938d-44be-9821-07337660ceef Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fb:3c:06:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fb:3c:06:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:51:a6:31:e1:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:12:09.829186 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.829176 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:12:09.829305 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.829292 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:12:09.832209 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.832184 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:12:09.832344 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.832212 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-225.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:12:09.832391 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.832346 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:12:09.832391 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.832355 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:12:09.832446 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.832390 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:09.832446 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.832404 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:09.833284 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.833274 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:12:09.837826 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.837814 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:12:09.837935 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.837926 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:12:09.838766 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.838756 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:12:09.838797 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.838770 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:12:09.838797 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.838782 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:12:09.838797 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.838790 2577 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:12:09.838901 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.838797 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:12:09.839105 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.839085 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:09.843877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.843860 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:12:09.849372 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.849357 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:09.853234 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.853220 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:12:09.853307 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.853252 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:12:09.853954 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.853943 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:12:09.854009 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.853959 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:12:09.854009 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.853979 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:12:09.854009 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.853995 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:12:09.854009 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854001 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:12:09.854009 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854008 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:12:09.854590 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854582 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:12:09.854621 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854590 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:12:09.854621 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854597 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:12:09.854621 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854603 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:12:09.854621 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854616 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:12:09.854817 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.854624 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:12:09.855316 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.855307 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:12:09.855347 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.855316 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:12:09.858926 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.858912 2577 server.go:1263] "Started kubelet" Nov 02 21:12:09.862134 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.859002 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:12:09.862218 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.861982 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:12:09.862218 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.862180 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:12:09.862949 ip-10-0-137-225 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:12:09.863444 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.863426 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:12:09.866416 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.866401 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:12:09.870024 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:09.870006 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 21:12:09.872312 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.872295 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:12:09.872406 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.872315 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:12:09.872461 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:09.872400 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:09.872906 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.872848 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:12:09.872906 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.872865 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:12:09.872906 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.872881 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:12:09.873103 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.872943 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:12:09.873103 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.872954 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:12:09.873302 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.873281 2577 factory.go:55] Registering systemd factory Nov 02 21:12:09.873302 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.873303 2577 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:12:09.873504 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.873490 2577 factory.go:153] Registering CRI-O factory Nov 02 21:12:09.873504 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.873505 2577 factory.go:223] Registration of the crio container factory successfully Nov 02 21:12:09.873610 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.873546 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:12:09.873610 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.873562 2577 factory.go:103] Registering Raw factory Nov 02 21:12:09.873610 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.873571 2577 manager.go:1199] Started watching for new ooms in manager Nov 02 21:12:09.874586 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.874569 2577 manager.go:319] Starting recovery of all containers Nov 02 21:12:09.887919 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.887904 2577 manager.go:324] Recovery completed Nov 02 21:12:09.895427 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.895411 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:09.897103 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.897084 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:09.897168 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.897119 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:09.897168 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.897134 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:09.897643 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.897630 2577 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:12:09.897643 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.897641 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:12:09.897744 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.897659 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:12:09.897744 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.897634 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:09.898424 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.898408 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:09.898481 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.898438 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:09.898481 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.898451 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:09.899769 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.899754 2577 policy_none.go:50] "Start" Nov 02 21:12:09.899769 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.899771 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:12:09.899868 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.899782 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:12:09.901720 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.901709 2577 policy_none.go:44] "Start" Nov 02 21:12:09.950682 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.950650 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:12:09.953615 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.953601 2577 manager.go:341] "Starting Device Plugin manager" Nov 02 21:12:09.953699 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:09.953639 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:12:09.953699 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.953651 2577 server.go:87] "Starting device plugin registration server" Nov 02 21:12:09.953889 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.953873 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:12:09.953992 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.953888 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:12:09.954068 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.953997 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:12:09.954068 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.954060 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:12:09.954068 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.954068 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:12:09.955434 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:09.955409 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:12:09.955517 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:09.955466 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:09.988222 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:09.988200 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-225.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 21:12:09.990780 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:09.988200 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-225.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 21:12:10.024252 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.024186 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:12:10.025387 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.025371 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:12:10.025477 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.025392 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:12:10.025477 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.025411 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:12:10.025477 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.025440 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 21:12:10.036158 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.035183 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-225.us-west-2.compute.internal.18e456befb38f866 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-225.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-225.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:12:09.85888983 +0000 UTC m=+0.425972062,LastTimestamp:2026-11-02 21:12:09.85888983 +0000 UTC m=+0.425972062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-225.us-west-2.compute.internal,}" Nov 02 21:12:10.048482 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.048458 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 21:12:10.048482 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.048473 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 21:12:10.048608 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.048493 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-225.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 21:12:10.053606 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.053592 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:12:10.053652 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.053632 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ghscb" Nov 02 21:12:10.054550 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.054538 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:10.055282 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.055267 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:10.055358 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.055298 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:10.055358 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.055313 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:10.055358 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.055344 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.126075 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.126037 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal"] Nov 02 21:12:10.126165 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.126108 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:10.127568 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.127553 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:10.127630 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.127584 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:10.127630 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.127595 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:10.129685 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.129672 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:10.129818 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.129800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.129888 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.129870 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:10.130320 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.130306 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:10.130384 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.130316 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:10.130384 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.130335 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:10.130384 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.130348 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:10.130496 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.130339 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:10.130496 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.130424 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:10.132384 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.132371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.132446 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.132394 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:10.133034 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.133020 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:10.133097 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.133044 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:10.133097 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.133058 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:10.153397 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.153379 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:10.156123 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.156108 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.156203 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.156133 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-225.us-west-2.compute.internal\": node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.164283 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.164267 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.166919 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.166901 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.174279 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.174263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.174359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.174286 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.174359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.174301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b27bf1c010fcac7e9afcd7fd0096491-config\") pod \"kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"1b27bf1c010fcac7e9afcd7fd0096491\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.275278 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.275221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.275278 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.275250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.275278 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.275266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b27bf1c010fcac7e9afcd7fd0096491-config\") pod \"kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"1b27bf1c010fcac7e9afcd7fd0096491\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.275437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.275293 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b27bf1c010fcac7e9afcd7fd0096491-config\") pod \"kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"1b27bf1c010fcac7e9afcd7fd0096491\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.275437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.275312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.275437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.275322 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.277764 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.277746 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.378452 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.378426 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.467448 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.467422 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.470577 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.470559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:10.478502 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.478480 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.579216 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.579045 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.679435 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.679403 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.780145 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.780115 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.793563 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.793542 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 21:12:10.793719 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:10.793700 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:12:10.793762 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.793671 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa4a48b711f844547b37f00e18a10669-41cfdfea4362ef04.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-225.us-west-2.compute.internal.18e456befd806adc\": read tcp 10.0.137.225:40054->3.150.79.186:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-225.us-west-2.compute.internal.18e456befd806adc default 4969 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-225.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-225.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-225.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:12:09 +0000 UTC,LastTimestamp:2026-11-02 21:12:10.12758923 +0000 UTC m=+0.694671465,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-225.us-west-2.compute.internal,}" Nov 02 21:12:10.880227 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.880164 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:10.981347 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:10.981319 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:11.049084 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.049062 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:11.049910 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.049893 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:11.050004 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.049924 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:11.050004 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.049938 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:11.059159 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.059132 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:07:10 +0000 UTC" deadline="2028-06-30 17:10:36.848720865 +0000 UTC" Nov 02 21:12:11.059159 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.059157 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14539h58m25.789567115s" Nov 02 21:12:11.081880 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:11.081859 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:11.163636 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.163573 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:11.182961 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:11.182939 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:11.215737 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.215716 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:11.270830 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.270808 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:11.273054 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.273030 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:11.337577 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.337555 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:11.338360 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.338348 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:11.400705 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.400687 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:11.747742 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:11.747706 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431b8dc08018bf9de47ad119a0b52d06.slice/crio-9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65 WatchSource:0}: Error finding container 9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65: Status 404 returned error can't find the container with id 9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65 Nov 02 21:12:11.748294 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:11.748268 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b27bf1c010fcac7e9afcd7fd0096491.slice/crio-557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55 WatchSource:0}: Error finding container 557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55: Status 404 returned error can't find the container with id 557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55 Nov 02 21:12:11.755199 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.755181 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:12:11.841933 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.841916 2577 apiserver.go:52] "Watching apiserver" Nov 02 21:12:11.875789 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.875740 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 21:12:11.953591 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.953572 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:12:11.954582 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.954564 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-4cd9w","kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-znl7n","kube-system/konnectivity-agent-h6889","openshift-multus/multus-s75ln","openshift-multus/network-metrics-daemon-2fcgs","openshift-ovn-kubernetes/ovnkube-node-wgl8r","kube-system/global-pull-secret-syncer-lsl4s","openshift-multus/multus-additional-cni-plugins-5zccx","openshift-network-diagnostics/network-check-target-j5vdc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5","openshift-cluster-node-tuning-operator/tuned-8v27x","openshift-dns/node-resolver-w8jd6"] Nov 02 21:12:11.958441 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.958426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:11.958505 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:11.958488 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:11.960535 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.960522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.962685 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.962671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:11.964976 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.964961 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:11.967329 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.967312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:11.969699 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.969682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.972116 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.972101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:11.972189 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:11.972160 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:11.976547 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.976519 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:11.979961 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.979941 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:11.982766 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.982747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:11.984564 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97770c2-375e-409d-9795-5cea674e8f2d-agent-certs\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:11.984637 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwnh\" (UniqueName: \"kubernetes.io/projected/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-kube-api-access-6nwnh\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:11.984637 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4rr\" (UniqueName: \"kubernetes.io/projected/fca08020-5c83-4f30-8ab3-054ba4bc6171-kube-api-access-5g4rr\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:11.984637 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-kubelet\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.984637 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-multus-certs\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.984802 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-etc-kubernetes\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.984802 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz26f\" (UniqueName: \"kubernetes.io/projected/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-kube-api-access-fz26f\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:11.984802 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-cnibin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.984802 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-netns\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985022 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-multus\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985022 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-hostroot\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985022 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-daemon-config\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985022 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-dbus\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:11.985022 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.984965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-modprobe-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:11.985086 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdps\" (UniqueName: \"kubernetes.io/projected/e2498872-da17-4526-8749-ce90d1f7fb7c-kube-api-access-tmdps\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fca08020-5c83-4f30-8ab3-054ba4bc6171-host-slash\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysconfig\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-kubernetes\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985206 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-systemd\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-run\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2498872-da17-4526-8749-ce90d1f7fb7c-serviceca\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-k8s-cni-cncf-io\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-lib-modules\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-tmp\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2498872-da17-4526-8749-ce90d1f7fb7c-host\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fca08020-5c83-4f30-8ab3-054ba4bc6171-iptables-alerter-script\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-socket-dir-parent\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-var-lib-kubelet\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-host\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-system-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-cni-binary-copy\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-conf\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-tuned\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8x8\" (UniqueName: \"kubernetes.io/projected/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-kube-api-access-2d8x8\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-hosts-file\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-tmp-dir\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97770c2-375e-409d-9795-5cea674e8f2d-konnectivity-ca\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-bin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j77w\" (UniqueName: \"kubernetes.io/projected/f35a604c-01b2-42fb-846e-a2d3f54045b3-kube-api-access-4j77w\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.985877 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-conf-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:11.986312 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-kubelet-config\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:11.986312 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:11.986312 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.985960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-sys\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:11.987568 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:11.987549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.017026 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.017004 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sdlzm\"" Nov 02 21:12:12.017148 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.017033 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:12.017148 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.017057 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:12.018743 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.018725 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:12:12.018743 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.018726 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:12:12.018845 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.018743 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z2vtc\"" Nov 02 21:12:12.020592 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.020575 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:12.029539 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.029506 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"1b27bf1c010fcac7e9afcd7fd0096491","Type":"ContainerStarted","Data":"557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55"} Nov 02 21:12:12.030561 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.030541 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"431b8dc08018bf9de47ad119a0b52d06","Type":"ContainerStarted","Data":"9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65"} Nov 02 21:12:12.057683 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057660 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:12.057786 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057749 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zzckx\"" Nov 02 21:12:12.057786 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057772 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:12:12.057875 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057780 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:12:12.057875 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057786 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6fz8\"" Nov 02 21:12:12.057875 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057832 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-q4vfp\"" Nov 02 21:12:12.057875 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057851 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:12:12.057875 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057873 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:12:12.058107 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.057928 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:12:12.058107 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.058006 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:12:12.058731 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.058708 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:12:12.058859 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.058735 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:12:12.058956 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.058768 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:12:12.063171 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063153 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ps8jp\"" Nov 02 21:12:12.063359 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063344 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:12:12.063509 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063486 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:12:12.063596 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063524 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:12:12.063596 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063526 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dtndn\"" Nov 02 21:12:12.063596 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063532 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:12:12.063596 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063568 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:12:12.063763 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063634 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:12:12.063916 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.063904 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:12:12.069253 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.069232 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c5zxb\"" Nov 02 21:12:12.069306 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.069268 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:12:12.069306 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.069276 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-crvzm\"" Nov 02 21:12:12.069392 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.069311 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:12:12.071306 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.071286 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:12:12.071306 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.071286 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:12:12.071431 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.071312 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-22r6w" Nov 02 21:12:12.071431 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.071289 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:12:12.071431 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.071349 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:12:12.074402 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.074387 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:12:12.087025 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-var-lib-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.087084 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.087084 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2498872-da17-4526-8749-ce90d1f7fb7c-host\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:12.087178 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fca08020-5c83-4f30-8ab3-054ba4bc6171-iptables-alerter-script\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:12.087178 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087178 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-socket-dir-parent\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087178 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087146 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2498872-da17-4526-8749-ce90d1f7fb7c-host\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-system-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-conf\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-systemd-units\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-ovn\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfg8h\" (UniqueName: \"kubernetes.io/projected/a8c954c9-200d-4823-b029-54cdebd356aa-kube-api-access-xfg8h\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-registration-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-system-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-socket-dir-parent\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-hosts-file\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:12.087353 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087349 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087373 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-conf\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-tmp-dir\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087399 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-hosts-file\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97770c2-375e-409d-9795-5cea674e8f2d-konnectivity-ca\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j77w\" (UniqueName: \"kubernetes.io/projected/f35a604c-01b2-42fb-846e-a2d3f54045b3-kube-api-access-4j77w\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-etc-selinux\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-conf-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-conf-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-kubelet-config\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-sys\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-kubelet-config\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-device-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fca08020-5c83-4f30-8ab3-054ba4bc6171-iptables-alerter-script\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-tmp-dir\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-sys-fs\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.087810 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.087706 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087712 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-sys\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxjm\" (UniqueName: \"kubernetes.io/projected/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kube-api-access-rsxjm\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4rr\" (UniqueName: \"kubernetes.io/projected/fca08020-5c83-4f30-8ab3-054ba4bc6171-kube-api-access-5g4rr\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.087766 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret podName:cbf2c02b-9cee-4a8b-a809-d2b52ec9235b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:12.587751536 +0000 UTC m=+3.154833762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret") pod "global-pull-secret-syncer-lsl4s" (UID: "cbf2c02b-9cee-4a8b-a809-d2b52ec9235b") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-multus-certs\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-log-socket\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087870 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-multus-certs\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087906 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-systemd\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-var-lib-kubelet\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz26f\" (UniqueName: \"kubernetes.io/projected/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-kube-api-access-fz26f\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.087972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-netns\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-multus\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-systemd\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-multus\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.088679 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088073 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-netns\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-daemon-config\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088120 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-var-lib-kubelet\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-modprobe-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-run\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088272 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-modprobe-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088281 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97770c2-375e-409d-9795-5cea674e8f2d-konnectivity-ca\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2498872-da17-4526-8749-ce90d1f7fb7c-serviceca\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-k8s-cni-cncf-io\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-lib-modules\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.088361 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-run\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-k8s-cni-cncf-io\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.088420 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:12.588402054 +0000 UTC m=+3.155484293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-node-log\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.089229 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-config\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-lib-modules\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-host\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088578 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-daemon-config\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-kubelet\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088629 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2498872-da17-4526-8749-ce90d1f7fb7c-serviceca\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-host\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-cni-binary-copy\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-tuned\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8x8\" (UniqueName: \"kubernetes.io/projected/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-kube-api-access-2d8x8\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c954c9-200d-4823-b029-54cdebd356aa-ovn-node-metrics-cert\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cnibin\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-bin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-systemd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.089914 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-bin\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-env-overrides\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-bin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-slash\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088916 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-script-lib\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97770c2-375e-409d-9795-5cea674e8f2d-agent-certs\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.088979 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwnh\" (UniqueName: \"kubernetes.io/projected/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-kube-api-access-6nwnh\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089118 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-cni-binary-copy\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-kubelet\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-etc-kubernetes\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-etc-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-netd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089268 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-etc-kubernetes\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090437 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-socket-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089347 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-kubelet\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-cnibin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-hostroot\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-dbus\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdczn\" (UniqueName: \"kubernetes.io/projected/d4f97caf-1c7a-484a-8b2d-a885eb56d793-kube-api-access-vdczn\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089462 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-cnibin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdps\" (UniqueName: \"kubernetes.io/projected/e2498872-da17-4526-8749-ce90d1f7fb7c-kube-api-access-tmdps\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089483 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-hostroot\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fca08020-5c83-4f30-8ab3-054ba4bc6171-host-slash\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysconfig\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-kubernetes\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-dbus\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089579 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fca08020-5c83-4f30-8ab3-054ba4bc6171-host-slash\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-tmp\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-netns\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089624 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysconfig\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.090929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-kubernetes\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.091466 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.089670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-system-cni-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.092472 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.092454 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-tmp\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.092549 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.092509 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-tuned\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.092724 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.092705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97770c2-375e-409d-9795-5cea674e8f2d-agent-certs\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:12.099274 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.099259 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:12.148555 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.148504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz26f\" (UniqueName: \"kubernetes.io/projected/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-kube-api-access-fz26f\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:12.148783 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.148762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4rr\" (UniqueName: \"kubernetes.io/projected/fca08020-5c83-4f30-8ab3-054ba4bc6171-kube-api-access-5g4rr\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:12.148891 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.148869 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j77w\" (UniqueName: \"kubernetes.io/projected/f35a604c-01b2-42fb-846e-a2d3f54045b3-kube-api-access-4j77w\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.149652 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.149633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8x8\" (UniqueName: \"kubernetes.io/projected/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-kube-api-access-2d8x8\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.150192 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.150172 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwnh\" (UniqueName: \"kubernetes.io/projected/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-kube-api-access-6nwnh\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:12.150271 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.150256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdps\" (UniqueName: \"kubernetes.io/projected/e2498872-da17-4526-8749-ce90d1f7fb7c-kube-api-access-tmdps\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:12.190159 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdczn\" (UniqueName: \"kubernetes.io/projected/d4f97caf-1c7a-484a-8b2d-a885eb56d793-kube-api-access-vdczn\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.190227 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-netns\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190227 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190227 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-system-cni-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.190334 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190242 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190334 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-netns\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190334 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190272 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-system-cni-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.190334 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-var-lib-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190334 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-systemd-units\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-ovn\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190387 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-var-lib-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190423 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-ovn\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190424 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-systemd-units\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfg8h\" (UniqueName: \"kubernetes.io/projected/a8c954c9-200d-4823-b029-54cdebd356aa-kube-api-access-xfg8h\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190466 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.190484 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-registration-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190536 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-registration-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-etc-selinux\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-device-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-sys-fs\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-device-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190692 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-etc-selinux\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxjm\" (UniqueName: \"kubernetes.io/projected/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kube-api-access-rsxjm\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-sys-fs\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-log-socket\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190785 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.190791 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-log-socket\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-node-log\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190851 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-config\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190906 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-node-log\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190868 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-kubelet\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.190993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c954c9-200d-4823-b029-54cdebd356aa-ovn-node-metrics-cert\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cnibin\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191053 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-systemd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191039 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-kubelet\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-bin\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cnibin\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.191330 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-env-overrides\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191131 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-systemd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191139 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-bin\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-slash\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-script-lib\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-etc-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191240 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-slash\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-netd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-socket-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191463 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-socket-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-etc-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-netd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191538 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-config\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-env-overrides\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191732 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.192602 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.191861 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-script-lib\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.192602 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.192085 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.193191 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.193171 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c954c9-200d-4823-b029-54cdebd356aa-ovn-node-metrics-cert\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.249454 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.249430 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:12.249454 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.249451 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:12.249614 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.249460 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8m6ts for pod openshift-network-diagnostics/network-check-target-j5vdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:12.249614 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.249534 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts podName:b7f9e303-8cb2-497a-98dc-c80092327b2b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:12.749519725 +0000 UTC m=+3.316601967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8m6ts" (UniqueName: "kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts") pod "network-check-target-j5vdc" (UID: "b7f9e303-8cb2-497a-98dc-c80092327b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:12.251560 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.251538 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxjm\" (UniqueName: \"kubernetes.io/projected/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kube-api-access-rsxjm\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.251837 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.251821 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfg8h\" (UniqueName: \"kubernetes.io/projected/a8c954c9-200d-4823-b029-54cdebd356aa-kube-api-access-xfg8h\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.252005 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.251975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdczn\" (UniqueName: \"kubernetes.io/projected/d4f97caf-1c7a-484a-8b2d-a885eb56d793-kube-api-access-vdczn\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.273285 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.273267 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:12.279797 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.279782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:12.283701 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.283677 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cba6681_dad1_4447_9c10_3eb9e98a7ebc.slice/crio-003d0261adff8b963561d0c5a9ba62b8a6013b4321e46fefaff1b8a33e5c8754 WatchSource:0}: Error finding container 003d0261adff8b963561d0c5a9ba62b8a6013b4321e46fefaff1b8a33e5c8754: Status 404 returned error can't find the container with id 003d0261adff8b963561d0c5a9ba62b8a6013b4321e46fefaff1b8a33e5c8754 Nov 02 21:12:12.288120 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.288102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:12.288816 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.288786 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b7f820_cae3_42ef_a19e_9c75d9323cbc.slice/crio-d2d1063ca9a922327272befe2debd015f25babb0f4b4aaac8ae3afd1158acd50 WatchSource:0}: Error finding container d2d1063ca9a922327272befe2debd015f25babb0f4b4aaac8ae3afd1158acd50: Status 404 returned error can't find the container with id d2d1063ca9a922327272befe2debd015f25babb0f4b4aaac8ae3afd1158acd50 Nov 02 21:12:12.294074 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.294053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:12.296696 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.296677 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca08020_5c83_4f30_8ab3_054ba4bc6171.slice/crio-d638416e67a97861c343008ee706cf3264e309f90080c300ef9a7dece064ada2 WatchSource:0}: Error finding container d638416e67a97861c343008ee706cf3264e309f90080c300ef9a7dece064ada2: Status 404 returned error can't find the container with id d638416e67a97861c343008ee706cf3264e309f90080c300ef9a7dece064ada2 Nov 02 21:12:12.298737 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.298715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s75ln" Nov 02 21:12:12.302663 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.302545 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97770c2_375e_409d_9795_5cea674e8f2d.slice/crio-36c403b809929d29073ecb136ae3ba48c650a713384a3d657483c1a503106645 WatchSource:0}: Error finding container 36c403b809929d29073ecb136ae3ba48c650a713384a3d657483c1a503106645: Status 404 returned error can't find the container with id 36c403b809929d29073ecb136ae3ba48c650a713384a3d657483c1a503106645 Nov 02 21:12:12.305498 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.305480 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:12.308171 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.308152 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35a604c_01b2_42fb_846e_a2d3f54045b3.slice/crio-8d6b9296626564dcd6a62bed0d396655a974fbdc253a6cc14a82477644044d61 WatchSource:0}: Error finding container 8d6b9296626564dcd6a62bed0d396655a974fbdc253a6cc14a82477644044d61: Status 404 returned error can't find the container with id 8d6b9296626564dcd6a62bed0d396655a974fbdc253a6cc14a82477644044d61 Nov 02 21:12:12.314659 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.314640 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2498872_da17_4526_8749_ce90d1f7fb7c.slice/crio-12cccc653695c2f8688e36fc1f8a0178b10b1ad07bffd7e40496e89b6c0ca281 WatchSource:0}: Error finding container 12cccc653695c2f8688e36fc1f8a0178b10b1ad07bffd7e40496e89b6c0ca281: Status 404 returned error can't find the container with id 12cccc653695c2f8688e36fc1f8a0178b10b1ad07bffd7e40496e89b6c0ca281 Nov 02 21:12:12.339272 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.339252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:12.340449 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.340433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:12.347677 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.347661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:12.348720 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.348693 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59c2bf5_87b9_4d66_9b73_deae3d541ddd.slice/crio-9945efe52ef2c589a75181da82dbe968ddd93c4b0d8005141c048a6762bf49f9 WatchSource:0}: Error finding container 9945efe52ef2c589a75181da82dbe968ddd93c4b0d8005141c048a6762bf49f9: Status 404 returned error can't find the container with id 9945efe52ef2c589a75181da82dbe968ddd93c4b0d8005141c048a6762bf49f9 Nov 02 21:12:12.351353 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.351334 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c954c9_200d_4823_b029_54cdebd356aa.slice/crio-1e268033f5eadeafbc34910c3704fc8874dc8998013ebeec5ced0f0bd7c4c9ae WatchSource:0}: Error finding container 1e268033f5eadeafbc34910c3704fc8874dc8998013ebeec5ced0f0bd7c4c9ae: Status 404 returned error can't find the container with id 1e268033f5eadeafbc34910c3704fc8874dc8998013ebeec5ced0f0bd7c4c9ae Nov 02 21:12:12.357288 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:12.357268 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f97caf_1c7a_484a_8b2d_a885eb56d793.slice/crio-e332a04295deadd7974c3e44e8eee36de9dccf7e4f8360cfc823b992011f1b1f WatchSource:0}: Error finding container e332a04295deadd7974c3e44e8eee36de9dccf7e4f8360cfc823b992011f1b1f: Status 404 returned error can't find the container with id e332a04295deadd7974c3e44e8eee36de9dccf7e4f8360cfc823b992011f1b1f Nov 02 21:12:12.594435 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.594358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:12.594435 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.594400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:12.594620 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.594509 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:12.594620 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.594565 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:13.594551717 +0000 UTC m=+4.161633953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:12.594620 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.594509 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:12.594620 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.594604 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret podName:cbf2c02b-9cee-4a8b-a809-d2b52ec9235b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:13.594591411 +0000 UTC m=+4.161673652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret") pod "global-pull-secret-syncer-lsl4s" (UID: "cbf2c02b-9cee-4a8b-a809-d2b52ec9235b") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:12.796136 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:12.796102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:12.796303 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.796276 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:12.796303 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.796297 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:12.796369 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.796308 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8m6ts for pod openshift-network-diagnostics/network-check-target-j5vdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:12.796404 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:12.796375 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts podName:b7f9e303-8cb2-497a-98dc-c80092327b2b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:13.796356539 +0000 UTC m=+4.363438775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m6ts" (UniqueName: "kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts") pod "network-check-target-j5vdc" (UID: "b7f9e303-8cb2-497a-98dc-c80092327b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:13.034661 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.034565 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w8jd6" event={"ID":"f8b7f820-cae3-42ef-a19e-9c75d9323cbc","Type":"ContainerStarted","Data":"d2d1063ca9a922327272befe2debd015f25babb0f4b4aaac8ae3afd1158acd50"} Nov 02 21:12:13.035927 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.035900 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8v27x" event={"ID":"3cba6681-dad1-4447-9c10-3eb9e98a7ebc","Type":"ContainerStarted","Data":"003d0261adff8b963561d0c5a9ba62b8a6013b4321e46fefaff1b8a33e5c8754"} Nov 02 21:12:13.037341 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.037316 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" event={"ID":"c59c2bf5-87b9-4d66-9b73-deae3d541ddd","Type":"ContainerStarted","Data":"9945efe52ef2c589a75181da82dbe968ddd93c4b0d8005141c048a6762bf49f9"} Nov 02 21:12:13.038713 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.038685 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s75ln" event={"ID":"f35a604c-01b2-42fb-846e-a2d3f54045b3","Type":"ContainerStarted","Data":"8d6b9296626564dcd6a62bed0d396655a974fbdc253a6cc14a82477644044d61"} Nov 02 21:12:13.040200 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.040171 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-znl7n" event={"ID":"fca08020-5c83-4f30-8ab3-054ba4bc6171","Type":"ContainerStarted","Data":"d638416e67a97861c343008ee706cf3264e309f90080c300ef9a7dece064ada2"} Nov 02 21:12:13.041485 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.041431 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerStarted","Data":"e332a04295deadd7974c3e44e8eee36de9dccf7e4f8360cfc823b992011f1b1f"} Nov 02 21:12:13.043134 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.043107 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"1e268033f5eadeafbc34910c3704fc8874dc8998013ebeec5ced0f0bd7c4c9ae"} Nov 02 21:12:13.044548 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.044522 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4cd9w" event={"ID":"e2498872-da17-4526-8749-ce90d1f7fb7c","Type":"ContainerStarted","Data":"12cccc653695c2f8688e36fc1f8a0178b10b1ad07bffd7e40496e89b6c0ca281"} Nov 02 21:12:13.045848 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.045824 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6889" event={"ID":"b97770c2-375e-409d-9795-5cea674e8f2d","Type":"ContainerStarted","Data":"36c403b809929d29073ecb136ae3ba48c650a713384a3d657483c1a503106645"} Nov 02 21:12:13.074920 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.074850 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:07:12 +0000 UTC" deadline="2028-06-06 00:43:59.425705743 +0000 UTC" Nov 02 21:12:13.074920 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.074918 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13947h31m46.350792377s" Nov 02 21:12:13.603777 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.603738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:13.603984 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.603913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:13.604118 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.604054 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:13.604192 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.604112 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:13.604192 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.604176 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret podName:cbf2c02b-9cee-4a8b-a809-d2b52ec9235b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:15.604154828 +0000 UTC m=+6.171237051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret") pod "global-pull-secret-syncer-lsl4s" (UID: "cbf2c02b-9cee-4a8b-a809-d2b52ec9235b") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:13.604301 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.604197 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:15.604186939 +0000 UTC m=+6.171269165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:13.806237 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:13.806199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:13.806433 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.806403 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:13.806433 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.806423 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:13.806548 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.806437 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8m6ts for pod openshift-network-diagnostics/network-check-target-j5vdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:13.806548 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:13.806500 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts podName:b7f9e303-8cb2-497a-98dc-c80092327b2b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:15.806481099 +0000 UTC m=+6.373563343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m6ts" (UniqueName: "kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts") pod "network-check-target-j5vdc" (UID: "b7f9e303-8cb2-497a-98dc-c80092327b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:14.027080 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:14.026996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:14.027236 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:14.027200 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:14.027356 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:14.027334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:14.027473 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:14.027453 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:14.027566 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:14.027552 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:14.027656 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:14.027639 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:14.075641 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:14.075579 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:07:12 +0000 UTC" deadline="2028-06-01 10:35:23.332551609 +0000 UTC" Nov 02 21:12:14.075641 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:14.075616 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13837h23m9.256940084s" Nov 02 21:12:15.622418 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:15.622381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:15.623068 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:15.622442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:15.623068 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.622617 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:15.623068 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.622685 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:19.622666013 +0000 UTC m=+10.189748247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:15.623068 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.622759 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:15.623068 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.622794 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret podName:cbf2c02b-9cee-4a8b-a809-d2b52ec9235b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:19.622783435 +0000 UTC m=+10.189865669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret") pod "global-pull-secret-syncer-lsl4s" (UID: "cbf2c02b-9cee-4a8b-a809-d2b52ec9235b") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:15.824216 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:15.823920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:15.824216 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.824107 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:15.824216 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.824125 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:15.824216 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.824134 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8m6ts for pod openshift-network-diagnostics/network-check-target-j5vdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:15.824216 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:15.824185 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts podName:b7f9e303-8cb2-497a-98dc-c80092327b2b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:19.824167795 +0000 UTC m=+10.391250029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m6ts" (UniqueName: "kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts") pod "network-check-target-j5vdc" (UID: "b7f9e303-8cb2-497a-98dc-c80092327b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:16.029398 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:16.028777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:16.029398 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:16.028907 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:16.029398 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:16.029036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:16.029398 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:16.029128 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:16.029398 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:16.029184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:16.029398 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:16.029308 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:17.080048 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:17.080012 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"1b27bf1c010fcac7e9afcd7fd0096491","Type":"ContainerStarted","Data":"4c6017d89730c7f3dadfb8e5ba00f009eef5ab8694da8cbbb0913fd9f8b47363"} Nov 02 21:12:17.085158 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:17.085128 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="f4dc3edbd8c7d164d67c1746df5bfb34939d0b5dba6abff863315a83c82309a3" exitCode=0 Nov 02 21:12:17.085337 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:17.085225 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerDied","Data":"f4dc3edbd8c7d164d67c1746df5bfb34939d0b5dba6abff863315a83c82309a3"} Nov 02 21:12:17.103479 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:17.103449 2577 generic.go:356] "Generic (PLEG): container finished" podID="431b8dc08018bf9de47ad119a0b52d06" containerID="30e00b67f68a307c2eb6fe763882b3aaa780991e22033666bec44357b421363e" exitCode=0 Nov 02 21:12:17.103589 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:17.103496 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"431b8dc08018bf9de47ad119a0b52d06","Type":"ContainerDied","Data":"30e00b67f68a307c2eb6fe763882b3aaa780991e22033666bec44357b421363e"} Nov 02 21:12:17.192898 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:17.192681 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" podStartSLOduration=6.192661484 podStartE2EDuration="6.192661484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:17.192088938 +0000 UTC m=+7.759171186" watchObservedRunningTime="2026-11-02 21:12:17.192661484 +0000 UTC m=+7.759743727" Nov 02 21:12:18.028070 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:18.028039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:18.028242 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:18.028159 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:18.028242 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:18.028211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:18.028354 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:18.028257 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:18.028354 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:18.028290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:18.028354 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:18.028329 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:18.109194 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:18.109160 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6889" event={"ID":"b97770c2-375e-409d-9795-5cea674e8f2d","Type":"ContainerStarted","Data":"9b9fe798b8e6b0fd7f95cc41af4c205f614561411f94dd2ab46888e776f2c85f"} Nov 02 21:12:18.112380 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:18.112354 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"431b8dc08018bf9de47ad119a0b52d06","Type":"ContainerStarted","Data":"4eaa27b6e7a9a6bea2cd2162cca66762ddd1098c61626cd5c87f05fe9b2d47f3"} Nov 02 21:12:18.338038 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:18.337935 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" podStartSLOduration=7.337923013 podStartE2EDuration="7.337923013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:18.337372162 +0000 UTC m=+8.904454403" watchObservedRunningTime="2026-11-02 21:12:18.337923013 +0000 UTC m=+8.905005255" Nov 02 21:12:18.338262 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:18.338088 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h6889" podStartSLOduration=3.314179275 podStartE2EDuration="8.338078769s" totalImagesPullingTime="5.023899494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:18.222663333 +0000 UTC m=+8.789745574" watchObservedRunningTime="2026-11-02 21:12:18.338078769 +0000 UTC m=+8.905161012" Nov 02 21:12:19.117321 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:19.116808 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4cd9w" event={"ID":"e2498872-da17-4526-8749-ce90d1f7fb7c","Type":"ContainerStarted","Data":"67673f4e5c0ec44936486e39c488d91b9d9c7290a800f24ce03b06c81d303ef0"} Nov 02 21:12:19.656055 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:19.656015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:19.656234 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:19.656089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:19.656326 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.656266 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:19.656398 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.656330 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:27.656311882 +0000 UTC m=+18.223394127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:19.656398 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.656389 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:19.656505 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.656421 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret podName:cbf2c02b-9cee-4a8b-a809-d2b52ec9235b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:27.656410342 +0000 UTC m=+18.223492575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret") pod "global-pull-secret-syncer-lsl4s" (UID: "cbf2c02b-9cee-4a8b-a809-d2b52ec9235b") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:19.857508 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:19.857474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:19.857677 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.857652 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:19.857677 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.857671 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:19.857775 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.857683 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8m6ts for pod openshift-network-diagnostics/network-check-target-j5vdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:19.857775 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:19.857737 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts podName:b7f9e303-8cb2-497a-98dc-c80092327b2b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:27.857719459 +0000 UTC m=+18.424801693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m6ts" (UniqueName: "kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts") pod "network-check-target-j5vdc" (UID: "b7f9e303-8cb2-497a-98dc-c80092327b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:20.025847 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:20.025782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:20.026020 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:20.025881 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:20.026082 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:20.026030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:20.026136 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:20.026111 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:20.026195 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:20.026179 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:20.026285 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:20.026263 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:20.124629 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:20.124485 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" event={"ID":"c59c2bf5-87b9-4d66-9b73-deae3d541ddd","Type":"ContainerStarted","Data":"0f4a471eaf20eb91fc02711eaf0103cb0f9b629bfaf44b03a7fa54f82c0fcb11"} Nov 02 21:12:20.197477 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:20.197445 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:20.198210 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:20.198181 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:20.310736 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:20.310249 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4cd9w" podStartSLOduration=4.276804203 podStartE2EDuration="10.310228278s" totalImagesPullingTime="6.033424075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:19.230321854 +0000 UTC m=+9.797404096" watchObservedRunningTime="2026-11-02 21:12:20.310228278 +0000 UTC m=+10.877310521" Nov 02 21:12:21.130559 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:21.130529 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s75ln" event={"ID":"f35a604c-01b2-42fb-846e-a2d3f54045b3","Type":"ContainerStarted","Data":"55766b4f7f3ca871eb874abc2ab91b64b67e9a03cbc4e63f040641dafccd0276"} Nov 02 21:12:21.285450 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:21.285426 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:12:21.962286 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:21.962142 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:12:21.285446461Z","UUID":"6be640c3-9223-423d-bcc3-a33f0b0a8290","Handler":null,"Name":"","Endpoint":""} Nov 02 21:12:21.963683 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:21.963661 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:12:21.963781 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:21.963691 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:12:22.026062 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.026032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:22.026182 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.026032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:22.026182 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:22.026150 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:22.026274 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.026047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:22.026380 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:22.026326 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:22.026380 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:22.026229 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:22.133625 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.133593 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="dec4b87b24086362247941c6f9c6ba16ace51e11de291a73c3a85a62dd3039c5" exitCode=0 Nov 02 21:12:22.134060 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.133672 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerDied","Data":"dec4b87b24086362247941c6f9c6ba16ace51e11de291a73c3a85a62dd3039c5"} Nov 02 21:12:22.136418 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.136202 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w8jd6" event={"ID":"f8b7f820-cae3-42ef-a19e-9c75d9323cbc","Type":"ContainerStarted","Data":"63c74661c23d34b3a270e6d4af25bd5f6f9c7a1275a060bef71981f7504a6410"} Nov 02 21:12:22.138728 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.138707 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" event={"ID":"c59c2bf5-87b9-4d66-9b73-deae3d541ddd","Type":"ContainerStarted","Data":"7758b22630227d964e1f76eff658833d86c5c73df09d627a0c3c33885a37da80"} Nov 02 21:12:22.250985 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:22.250860 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-s75ln" podStartSLOduration=3.666926297 podStartE2EDuration="12.250842585s" totalImagesPullingTime="8.583916288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:21.243944093 +0000 UTC m=+11.811026335" watchObservedRunningTime="2026-11-02 21:12:22.250842585 +0000 UTC m=+12.817924828" Nov 02 21:12:23.142992 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:23.142898 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-znl7n" event={"ID":"fca08020-5c83-4f30-8ab3-054ba4bc6171","Type":"ContainerStarted","Data":"6deda5472f164c0f4666c9b070b5ef834aeb3cf17a31aff23865c4f4d7203e4f"} Nov 02 21:12:23.145802 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:23.145770 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" event={"ID":"c59c2bf5-87b9-4d66-9b73-deae3d541ddd","Type":"ContainerStarted","Data":"9c45fe82ffbf682169e3c5a3c1277f7460bcb536797bdce9c9b8275a4c4c1a23"} Nov 02 21:12:23.255114 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:23.255060 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-w8jd6" podStartSLOduration=4.459713723 podStartE2EDuration="13.255046653s" totalImagesPullingTime="8.79533293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:22.363657604 +0000 UTC m=+12.930739848" watchObservedRunningTime="2026-11-02 21:12:23.255046653 +0000 UTC m=+13.822128895" Nov 02 21:12:23.371042 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:23.370999 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-znl7n" podStartSLOduration=4.37311918 podStartE2EDuration="13.370985287s" totalImagesPullingTime="8.997866107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:23.254542857 +0000 UTC m=+13.821625100" watchObservedRunningTime="2026-11-02 21:12:23.370985287 +0000 UTC m=+13.938067523" Nov 02 21:12:23.371639 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:23.371604 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" podStartSLOduration=3.179764046 podStartE2EDuration="13.371597004s" totalImagesPullingTime="10.191832958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:23.370947963 +0000 UTC m=+13.938030218" watchObservedRunningTime="2026-11-02 21:12:23.371597004 +0000 UTC m=+13.938679245" Nov 02 21:12:24.025998 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:24.025901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:24.025998 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:24.025907 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:24.026197 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:24.026046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:24.026197 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:24.026064 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:24.026197 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:24.026150 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:24.026340 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:24.026228 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:24.150534 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:24.150494 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="bd2a209e2d892a435a3733dbb80bad08ddbcbd11ed90ed3667e46791910d8229" exitCode=0 Nov 02 21:12:24.151099 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:24.150579 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerDied","Data":"bd2a209e2d892a435a3733dbb80bad08ddbcbd11ed90ed3667e46791910d8229"} Nov 02 21:12:24.152317 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:24.152290 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8v27x" event={"ID":"3cba6681-dad1-4447-9c10-3eb9e98a7ebc","Type":"ContainerStarted","Data":"72a94d41eee73f89495370fa32f729366579d7303dc15c4e9920f3139e0392c6"} Nov 02 21:12:24.379446 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:24.379342 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8v27x" podStartSLOduration=3.162159454 podStartE2EDuration="14.379327414s" totalImagesPullingTime="11.21716796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:24.378903677 +0000 UTC m=+14.945985921" watchObservedRunningTime="2026-11-02 21:12:24.379327414 +0000 UTC m=+14.946409655" Nov 02 21:12:26.028732 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:26.028705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:26.028732 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:26.028726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:26.029491 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:26.028743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:26.029491 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:26.028833 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:26.029491 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:26.028917 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:26.029491 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:26.029009 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:26.158944 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:26.158910 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="7912e5d2372dfe1df92e6c829b2f9a86b5d4f717813ab1888a9e062a192e91c5" exitCode=0 Nov 02 21:12:26.159122 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:26.158976 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerDied","Data":"7912e5d2372dfe1df92e6c829b2f9a86b5d4f717813ab1888a9e062a192e91c5"} Nov 02 21:12:27.722421 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:27.722259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:27.722779 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:27.722443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:27.722779 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.722367 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:27.722779 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.722573 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret podName:cbf2c02b-9cee-4a8b-a809-d2b52ec9235b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:43.722560207 +0000 UTC m=+34.289642427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret") pod "global-pull-secret-syncer-lsl4s" (UID: "cbf2c02b-9cee-4a8b-a809-d2b52ec9235b") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:27.722779 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.722525 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:27.722779 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.722606 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:43.722599623 +0000 UTC m=+34.289681842 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:27.923904 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:27.923698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:27.924065 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.923827 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:27.924065 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.924033 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:27.924065 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.924047 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8m6ts for pod openshift-network-diagnostics/network-check-target-j5vdc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:27.924202 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:27.924099 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts podName:b7f9e303-8cb2-497a-98dc-c80092327b2b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:43.92408193 +0000 UTC m=+34.491164166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m6ts" (UniqueName: "kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts") pod "network-check-target-j5vdc" (UID: "b7f9e303-8cb2-497a-98dc-c80092327b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:28.025875 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.025784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:28.025875 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.025824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:28.026069 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.025986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:28.026069 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:28.025979 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:28.026148 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:28.026075 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:28.026178 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:28.026164 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:28.166191 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166164 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:12:28.166561 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166535 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8c954c9-200d-4823-b029-54cdebd356aa" containerID="2820445b4802e0e96ef756754db20288c61f941605da1aaa8f8c386ee1aa852c" exitCode=1 Nov 02 21:12:28.166698 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166579 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"9d55d2020bd57ec827b2e87a7ce51e6f86617e120bb9fe8de6879a3cf279f26c"} Nov 02 21:12:28.166698 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166635 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"2dff2e7d8eff6cd4441b3e0e9f1fcb096cbd660ae47947604933ab8beea781b3"} Nov 02 21:12:28.166698 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166664 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"e27b70e139b3526b23a204930bcba6e2da6f7104a15d00a323162288daa720ab"} Nov 02 21:12:28.166698 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166693 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"b861d593e593fddc40c0b9e948efab1473c1f2eef92c740280d2bb8b1f823707"} Nov 02 21:12:28.166889 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166736 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerDied","Data":"2820445b4802e0e96ef756754db20288c61f941605da1aaa8f8c386ee1aa852c"} Nov 02 21:12:28.166889 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:28.166767 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"cba44c044eb4c141038c382a20109c9a37b14deecb753c2052946bcf9baf4e2a"} Nov 02 21:12:29.170788 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:29.170754 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="fd07488fd7c9d9495a0558bcb08deef892188d358fd0d93f7112f3dc01bce5f5" exitCode=0 Nov 02 21:12:29.171261 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:29.170821 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerDied","Data":"fd07488fd7c9d9495a0558bcb08deef892188d358fd0d93f7112f3dc01bce5f5"} Nov 02 21:12:30.026636 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:30.026610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:30.026787 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:30.026708 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:30.026787 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:30.026722 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:30.026787 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:30.026746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:30.026941 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:30.026844 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:30.026941 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:30.026931 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:30.175326 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:30.175296 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="695e9a32f480399d7233e29900912bbc0996d58416e685f3900e0ab7eba31c31" exitCode=0 Nov 02 21:12:30.175822 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:30.175344 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerDied","Data":"695e9a32f480399d7233e29900912bbc0996d58416e685f3900e0ab7eba31c31"} Nov 02 21:12:30.178303 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:30.178287 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:12:30.178647 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:30.178629 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"e03ce4d51190950d76af5b17238d4353021f9771ef58a72c434543fd11249a6c"} Nov 02 21:12:31.183578 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:31.183548 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zccx" event={"ID":"d4f97caf-1c7a-484a-8b2d-a885eb56d793","Type":"ContainerStarted","Data":"93a0908a97b5deb662d145cc79514045e0c0d800676bb882c9868aee223fe577"} Nov 02 21:12:31.300859 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:31.300810 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5zccx" podStartSLOduration=8.311554454 podStartE2EDuration="21.300796723s" totalImagesPullingTime="12.989242269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:31.300387321 +0000 UTC m=+21.867469562" watchObservedRunningTime="2026-11-02 21:12:31.300796723 +0000 UTC m=+21.867878965" Nov 02 21:12:31.343663 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:31.343634 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:31.343783 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:31.343770 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:12:31.344215 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:31.344195 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:32.025779 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:32.025741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:32.025985 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:32.025859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:32.025985 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:32.025749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:32.025985 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:32.025911 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:32.026401 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:32.026081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:32.026401 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:32.026126 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:32.188950 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:32.188791 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:12:32.189494 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:32.189342 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"0e6002994d6d8daeb8e0713ff027af3b119188e0d638f627a49ff580b7f82858"} Nov 02 21:12:32.189772 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:32.189752 2577 scope.go:122] "RemoveContainer" containerID="2820445b4802e0e96ef756754db20288c61f941605da1aaa8f8c386ee1aa852c" Nov 02 21:12:33.194893 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.194867 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:12:33.195311 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.195229 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" event={"ID":"a8c954c9-200d-4823-b029-54cdebd356aa","Type":"ContainerStarted","Data":"4cea6e94d98895aae0172670945485c1cccb58b1dd7b8d761579d0a7770b0f25"} Nov 02 21:12:33.195444 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.195429 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:33.195584 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.195565 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:33.195651 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.195639 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:33.209959 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.209941 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:33.210161 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.210063 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:33.467581 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:33.467492 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" podStartSLOduration=8.367943521 podStartE2EDuration="23.467479227s" totalImagesPullingTime="15.099535706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:33.416985855 +0000 UTC m=+23.984068091" watchObservedRunningTime="2026-11-02 21:12:33.467479227 +0000 UTC m=+24.034561468" Nov 02 21:12:34.026363 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.026334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:34.026512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.026333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:34.026512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.026449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:34.026512 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:34.026445 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:34.026652 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:34.026555 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:34.026685 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:34.026651 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:34.748847 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.748781 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsl4s"] Nov 02 21:12:34.749377 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.748895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:34.749377 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:34.749025 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:34.752288 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.752267 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2fcgs"] Nov 02 21:12:34.752388 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.752363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:34.752482 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:34.752465 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:34.752936 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.752917 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5vdc"] Nov 02 21:12:34.753012 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:34.753001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:34.753083 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:34.753067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:36.025778 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:36.025748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:36.026166 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:36.025862 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:36.026166 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:36.025934 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:36.026166 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:36.026039 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:37.026168 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:37.026080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:37.026608 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:37.026205 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:38.025988 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:38.025942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:38.026183 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:38.026092 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:38.026183 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:38.026142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:38.026568 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:38.026299 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:39.026577 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:39.026505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:39.026923 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:39.026617 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5vdc" podUID="b7f9e303-8cb2-497a-98dc-c80092327b2b" Nov 02 21:12:40.026508 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.026348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:40.026683 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:40.026561 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fcgs" podUID="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" Nov 02 21:12:40.026683 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.026420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:40.026683 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:40.026618 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsl4s" podUID="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" Nov 02 21:12:40.713768 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.713734 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeReady" Nov 02 21:12:40.713916 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.713906 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:12:40.826219 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.826189 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s95mt"] Nov 02 21:12:40.828207 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.828192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:40.876512 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.876448 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9cc7j"] Nov 02 21:12:40.878065 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.878051 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s95mt"] Nov 02 21:12:40.878117 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.878079 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9cc7j"] Nov 02 21:12:40.878150 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.878143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:40.880632 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.880607 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:12:40.880632 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.880618 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dfd2\"" Nov 02 21:12:40.880792 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.880635 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:12:40.918535 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.918510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-tmp-dir\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:40.918631 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.918537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbm7\" (UniqueName: \"kubernetes.io/projected/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-kube-api-access-vgbm7\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:40.918631 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.918578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:40.918714 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.918698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-config-volume\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:40.981812 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.981784 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:12:40.981812 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.981802 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:12:40.981949 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.981833 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jzn4w\"" Nov 02 21:12:40.981949 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:40.981840 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:12:41.019137 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-cert\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:41.019232 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-tmp-dir\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.019232 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbm7\" (UniqueName: \"kubernetes.io/projected/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-kube-api-access-vgbm7\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.019301 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.019301 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019272 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4b5\" (UniqueName: \"kubernetes.io/projected/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-kube-api-access-jz4b5\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:41.019363 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:41.019330 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:41.019363 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-config-volume\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.019432 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:41.019377 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:41.519362037 +0000 UTC m=+32.086444259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:41.019432 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019418 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-tmp-dir\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.019809 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.019793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-config-volume\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.026105 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.026091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:41.077219 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.077197 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbm7\" (UniqueName: \"kubernetes.io/projected/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-kube-api-access-vgbm7\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.082398 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.082380 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:12:41.082471 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.082444 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:12:41.082829 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.082815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nvsgt\"" Nov 02 21:12:41.119704 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.119681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4b5\" (UniqueName: \"kubernetes.io/projected/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-kube-api-access-jz4b5\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:41.119784 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.119731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-cert\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:41.121770 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.121749 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-cert\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:41.175716 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.175670 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4b5\" (UniqueName: \"kubernetes.io/projected/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-kube-api-access-jz4b5\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:41.190438 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.190417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:41.365200 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.365171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:41.522720 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.522653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:41.522836 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:41.522780 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:41.522873 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:41.522835 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:42.522822939 +0000 UTC m=+33.089905160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:41.531075 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.531046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:41.570448 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:41.570427 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9cc7j"] Nov 02 21:12:41.576285 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:41.576262 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5e5ce7_b1e0_46a6_8170_c379721e6e45.slice/crio-20490739847bd2cb2a505698ba4b350dc173ad5f97f98481fc509b0bfc990d70 WatchSource:0}: Error finding container 20490739847bd2cb2a505698ba4b350dc173ad5f97f98481fc509b0bfc990d70: Status 404 returned error can't find the container with id 20490739847bd2cb2a505698ba4b350dc173ad5f97f98481fc509b0bfc990d70 Nov 02 21:12:42.026446 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.026413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:42.026634 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.026614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:42.078963 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.078939 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:12:42.079251 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.079094 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mnft8\"" Nov 02 21:12:42.079251 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.079099 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r4znh\"" Nov 02 21:12:42.079251 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.079154 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:12:42.213487 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.213462 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9cc7j" event={"ID":"1f5e5ce7-b1e0-46a6-8170-c379721e6e45","Type":"ContainerStarted","Data":"20490739847bd2cb2a505698ba4b350dc173ad5f97f98481fc509b0bfc990d70"} Nov 02 21:12:42.530596 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:42.530566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:42.530724 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:42.530683 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:42.530764 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:42.530735 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:44.530722957 +0000 UTC m=+35.097805178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:43.739101 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:43.739062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:43.739559 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:43.739113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:43.739559 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:43.739232 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:43.739559 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:43.739306 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:13:15.739287278 +0000 UTC m=+66.306369518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : secret "metrics-daemon-secret" not found Nov 02 21:12:43.742091 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:43.742064 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:43.930047 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:43.930024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:43.930515 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:43.930486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:43.940443 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:43.940415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:43.943122 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:43.943096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:44.035575 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:44.035546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:44.046730 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:44.046699 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:44.166332 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:44.166293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:44.314777 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:44.314751 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsl4s"] Nov 02 21:12:44.319957 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:44.319932 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf2c02b_9cee_4a8b_a809_d2b52ec9235b.slice/crio-714882cb48529c99c446f53b0c6e8b684b9490fcd9f278f8874c5cec8a0f2836 WatchSource:0}: Error finding container 714882cb48529c99c446f53b0c6e8b684b9490fcd9f278f8874c5cec8a0f2836: Status 404 returned error can't find the container with id 714882cb48529c99c446f53b0c6e8b684b9490fcd9f278f8874c5cec8a0f2836 Nov 02 21:12:44.428357 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:44.428322 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5vdc"] Nov 02 21:12:44.433309 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:44.433279 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f9e303_8cb2_497a_98dc_c80092327b2b.slice/crio-cf657ea95a67c9ec705b9743bba8e0265b013d3362b47e45f4f5d93c6cfa39e6 WatchSource:0}: Error finding container cf657ea95a67c9ec705b9743bba8e0265b013d3362b47e45f4f5d93c6cfa39e6: Status 404 returned error can't find the container with id cf657ea95a67c9ec705b9743bba8e0265b013d3362b47e45f4f5d93c6cfa39e6 Nov 02 21:12:44.544261 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:44.544221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:44.544430 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:44.544380 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:44.544487 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:44.544454 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:48.544437604 +0000 UTC m=+39.111519840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:45.222572 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:45.222484 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9cc7j" event={"ID":"1f5e5ce7-b1e0-46a6-8170-c379721e6e45","Type":"ContainerStarted","Data":"91bcf05dfe6540f53410e0b11ac2b523fbcea9021af840d8bcc3048706055df6"} Nov 02 21:12:45.223725 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:45.223685 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5vdc" event={"ID":"b7f9e303-8cb2-497a-98dc-c80092327b2b","Type":"ContainerStarted","Data":"cf657ea95a67c9ec705b9743bba8e0265b013d3362b47e45f4f5d93c6cfa39e6"} Nov 02 21:12:45.224765 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:45.224743 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsl4s" event={"ID":"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b","Type":"ContainerStarted","Data":"714882cb48529c99c446f53b0c6e8b684b9490fcd9f278f8874c5cec8a0f2836"} Nov 02 21:12:45.335192 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:45.335141 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9cc7j" podStartSLOduration=1.93959879 podStartE2EDuration="5.33512908s" totalImagesPullingTime="3.39553029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:45.334534741 +0000 UTC m=+35.901616984" watchObservedRunningTime="2026-11-02 21:12:45.33512908 +0000 UTC m=+35.902211321" Nov 02 21:12:47.486270 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.486196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:47.556130 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.556098 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v8947"] Nov 02 21:12:47.589677 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.589447 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.613368 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.613327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:47.641820 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.641775 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v8947"] Nov 02 21:12:47.667390 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.667353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xb4t\" (UniqueName: \"kubernetes.io/projected/585fc755-2964-49cf-a948-e99760facfa2-kube-api-access-7xb4t\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.667571 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.667415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585fc755-2964-49cf-a948-e99760facfa2-data-volume\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.667571 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.667449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585fc755-2964-49cf-a948-e99760facfa2-crio-socket\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.667571 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.667478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.667571 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.667535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.716989 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.716938 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:12:47.717169 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.717022 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:12:47.717169 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.717054 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:12:47.727658 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.727629 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:12:47.727798 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.727717 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhk5v\"" Nov 02 21:12:47.751693 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.751607 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9"] Nov 02 21:12:47.768345 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.768311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xb4t\" (UniqueName: \"kubernetes.io/projected/585fc755-2964-49cf-a948-e99760facfa2-kube-api-access-7xb4t\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.768460 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.768357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585fc755-2964-49cf-a948-e99760facfa2-data-volume\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.768460 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.768377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585fc755-2964-49cf-a948-e99760facfa2-crio-socket\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.768460 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.768410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.768460 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.768443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.768644 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:47.768539 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:47.768644 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:47.768606 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:48.268587141 +0000 UTC m=+38.835669378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : configmap "kube-rbac-proxy" not found Nov 02 21:12:47.769260 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:47.769083 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:47.769260 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:47.769142 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:48.269124595 +0000 UTC m=+38.836206826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:47.769260 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.769215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585fc755-2964-49cf-a948-e99760facfa2-crio-socket\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.769260 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.769226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585fc755-2964-49cf-a948-e99760facfa2-data-volume\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.773142 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.773123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" Nov 02 21:12:47.775250 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.775229 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:47.807727 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.807497 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9"] Nov 02 21:12:47.827908 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.827885 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tjjtb\"" Nov 02 21:12:47.828098 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.828082 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xb4t\" (UniqueName: \"kubernetes.io/projected/585fc755-2964-49cf-a948-e99760facfa2-kube-api-access-7xb4t\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:47.869461 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.869431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hncfz\" (UniqueName: \"kubernetes.io/projected/319bc936-3891-49ed-81b9-a19a76825441-kube-api-access-hncfz\") pod \"migrator-66df5d496b-t82b9\" (UID: \"319bc936-3891-49ed-81b9-a19a76825441\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" Nov 02 21:12:47.887392 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.887343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:47.917298 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.917272 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:12:47.917423 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.917296 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:47.970719 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:47.970691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncfz\" (UniqueName: \"kubernetes.io/projected/319bc936-3891-49ed-81b9-a19a76825441-kube-api-access-hncfz\") pod \"migrator-66df5d496b-t82b9\" (UID: \"319bc936-3891-49ed-81b9-a19a76825441\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" Nov 02 21:12:48.001178 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.001144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:48.030241 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.030177 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncfz\" (UniqueName: \"kubernetes.io/projected/319bc936-3891-49ed-81b9-a19a76825441-kube-api-access-hncfz\") pod \"migrator-66df5d496b-t82b9\" (UID: \"319bc936-3891-49ed-81b9-a19a76825441\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" Nov 02 21:12:48.099010 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.098962 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" Nov 02 21:12:48.119941 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.119889 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:48.234204 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.234144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:48.273463 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.273428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:48.273632 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.273526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:48.273632 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:48.273560 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:48.273730 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:48.273641 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:49.273617611 +0000 UTC m=+39.840699851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : configmap "kube-rbac-proxy" not found Nov 02 21:12:48.273730 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:48.273649 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:48.273730 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:48.273695 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:49.273678181 +0000 UTC m=+39.840760401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:48.379521 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.379486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:48.489190 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.489157 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9"] Nov 02 21:12:48.494135 ip-10-0-137-225 kubenswrapper[2577]: W1102 21:12:48.494107 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319bc936_3891_49ed_81b9_a19a76825441.slice/crio-d32188a08fc267b64ca9ababa519fa63b542a085c78735f30a5ca0d523fe886b WatchSource:0}: Error finding container d32188a08fc267b64ca9ababa519fa63b542a085c78735f30a5ca0d523fe886b: Status 404 returned error can't find the container with id d32188a08fc267b64ca9ababa519fa63b542a085c78735f30a5ca0d523fe886b Nov 02 21:12:48.502849 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.502810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:48.576355 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:48.576329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:48.576542 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:48.576504 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:48.576609 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:48.576587 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:56.576564346 +0000 UTC m=+47.143646581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:49.238087 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:49.238034 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" event={"ID":"319bc936-3891-49ed-81b9-a19a76825441","Type":"ContainerStarted","Data":"d32188a08fc267b64ca9ababa519fa63b542a085c78735f30a5ca0d523fe886b"} Nov 02 21:12:49.239854 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:49.239820 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5vdc" event={"ID":"b7f9e303-8cb2-497a-98dc-c80092327b2b","Type":"ContainerStarted","Data":"c51e6e67fa5eabcfcd418eea8af845b932545d2b979a9141f0ef558305a8b703"} Nov 02 21:12:49.240009 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:49.239987 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:49.280860 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:49.280826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:49.281005 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:49.280918 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:49.281070 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:49.281018 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:51.281000712 +0000 UTC m=+41.848082950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : configmap "kube-rbac-proxy" not found Nov 02 21:12:49.281070 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:49.281059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:49.281212 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:49.281197 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:49.281264 ip-10-0-137-225 kubenswrapper[2577]: E1102 21:12:49.281250 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:51.28123664 +0000 UTC m=+41.848318864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:49.353146 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:49.353097 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j5vdc" podStartSLOduration=35.11779271 podStartE2EDuration="39.353085257s" totalImagesPullingTime="4.235292547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:49.352389359 +0000 UTC m=+39.919471601" watchObservedRunningTime="2026-11-02 21:12:49.353085257 +0000 UTC m=+39.920167498" Nov 02 21:12:49.595125 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:49.595066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:50.104863 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:50.104825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:50.245094 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:50.245059 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsl4s" event={"ID":"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b","Type":"ContainerStarted","Data":"c352bc0036f36875ffbecdd47d963d954bece656e13513307ef44e59ccc18979"} Nov 02 21:12:50.357929 ip-10-0-137-225 kubenswrapper[2577]: I1102 21:12:50.357881 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lsl4s" podStartSLOduration=34.657051917 podStartE2EDuration="40.357867351s" totalImagesPullingTime="5.700815434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:50.357098863 +0000 UTC m=+40.924181105" watchObservedRunningTime="2026-11-02 21:12:50.357867351 +0000 UTC m=+40.924949590" Nov 02 21:12:50.372103 ip-10-0-137-225 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 21:12:50.391357 ip-10-0-137-225 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 21:12:50.391515 ip-10-0-137-225 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 21:12:50.391559 ip-10-0-137-225 systemd[1]: kubelet.service: Consumed 2.246s CPU time. Nov 02 21:12:50.407672 ip-10-0-137-225 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:12:50.495916 ip-10-0-137-225 kubenswrapper[4441]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:50.495916 ip-10-0-137-225 kubenswrapper[4441]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:12:50.495916 ip-10-0-137-225 kubenswrapper[4441]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:50.495916 ip-10-0-137-225 kubenswrapper[4441]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:50.498030 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.497937 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.498030 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498029 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.498030 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498034 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498037 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498040 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498043 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498045 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498048 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498051 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498053 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498056 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498059 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498062 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498065 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498071 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498074 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498077 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498080 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498084 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498087 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498091 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.498123 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498093 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498097 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498099 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498102 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498106 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498110 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498113 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498116 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498118 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498121 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498124 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498127 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498130 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498133 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498136 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498138 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498141 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498144 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498146 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.498606 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498149 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498151 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498154 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498156 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498159 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498161 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498164 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498167 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498169 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498172 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498175 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498178 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498181 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498183 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498186 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498189 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498192 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498195 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498197 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498200 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.499082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498203 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498205 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498208 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498210 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498213 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498215 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498218 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498221 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498223 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498226 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498228 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498231 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498234 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498236 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498239 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498241 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498244 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498246 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498249 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498252 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.499578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498255 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498257 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498260 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498262 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498265 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498267 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498270 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498273 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498276 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498279 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498281 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498284 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498287 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498289 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498292 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498294 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498297 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498299 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498302 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.500095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498304 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498307 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498311 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498313 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498316 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498318 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.498321 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499096 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499103 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499106 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499109 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499112 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499115 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499118 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499121 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499124 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499127 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499130 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499132 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499135 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.500578 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499138 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499141 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499144 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499146 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499149 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499152 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499154 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499157 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499159 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499162 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499164 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499167 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499170 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499173 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499175 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499178 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499180 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499183 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499185 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.501085 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499188 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499190 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499193 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499196 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499198 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499200 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499203 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499206 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499208 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499212 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499215 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499217 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499220 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499224 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499228 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499232 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499235 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499238 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499240 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.501540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499243 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499245 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499248 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499250 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499253 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499256 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499258 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499260 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499264 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499266 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499270 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499272 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499275 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499277 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499280 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499282 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499285 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499287 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499290 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499293 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.502066 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499295 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499303 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499306 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499309 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499311 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499314 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499317 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499319 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499322 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499324 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499327 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499329 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499332 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499335 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499339 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499342 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499346 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499348 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499351 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.502575 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499354 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499356 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499359 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499362 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499364 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499366 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499369 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499371 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499374 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499377 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499379 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499382 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499385 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499388 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499391 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499394 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499834 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499839 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499842 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.503051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499845 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499847 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499850 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499852 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499855 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499858 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499860 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499863 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499866 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499868 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499871 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499873 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499877 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499882 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499885 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499889 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499892 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499895 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499898 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.503531 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499901 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499903 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499906 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499909 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499911 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499914 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499916 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499919 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499921 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499924 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499927 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499930 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499932 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499935 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499937 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499940 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499942 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499944 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499947 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.504234 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499949 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499952 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499954 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499957 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499960 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499962 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499978 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499983 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499986 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499989 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499992 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499994 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499997 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.499999 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500002 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500004 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500007 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500010 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500013 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500016 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.504985 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500018 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500021 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500023 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500026 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500029 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500031 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500034 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500036 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500039 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500041 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500044 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500046 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500049 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500051 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500054 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500056 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500059 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500061 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500064 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.505492 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500068 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500073 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500075 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500078 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500081 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500083 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500086 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500088 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500091 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500093 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500096 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500099 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500101 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500104 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500107 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500109 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500112 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500115 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500130 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500134 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.505956 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500137 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500140 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500143 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500145 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500149 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500151 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500232 4441 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500241 4441 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500247 4441 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500252 4441 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500256 4441 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500260 4441 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500265 4441 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500271 4441 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500274 4441 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500277 4441 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500281 4441 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500284 4441 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500287 4441 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500290 4441 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500293 4441 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500296 4441 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500299 4441 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:12:50.506471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500302 4441 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500306 4441 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500309 4441 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500312 4441 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500315 4441 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500319 4441 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500323 4441 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500326 4441 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500330 4441 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500333 4441 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500336 4441 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500339 4441 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500342 4441 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500345 4441 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500348 4441 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500352 4441 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500355 4441 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500358 4441 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500361 4441 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500364 4441 flags.go:64] FLAG: --enable-server="true" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500367 4441 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500372 4441 flags.go:64] FLAG: --event-burst="100" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500377 4441 flags.go:64] FLAG: --event-qps="50" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500380 4441 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:12:50.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500383 4441 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500386 4441 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500390 4441 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500393 4441 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500396 4441 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500399 4441 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500402 4441 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500405 4441 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500408 4441 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500411 4441 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500414 4441 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500417 4441 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500420 4441 flags.go:64] FLAG: --feature-gates="" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500424 4441 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500427 4441 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500430 4441 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500434 4441 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500437 4441 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500440 4441 flags.go:64] FLAG: --help="false" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500443 4441 flags.go:64] FLAG: --hostname-override="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500446 4441 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500449 4441 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500452 4441 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:12:50.507658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500456 4441 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500459 4441 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500462 4441 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500465 4441 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500468 4441 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500471 4441 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500475 4441 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500479 4441 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500482 4441 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500485 4441 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500488 4441 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500491 4441 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500494 4441 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500497 4441 flags.go:64] FLAG: --lock-file="" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500500 4441 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500503 4441 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500506 4441 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500511 4441 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500514 4441 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500517 4441 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500528 4441 flags.go:64] FLAG: --logging-format="text" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500531 4441 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500535 4441 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500537 4441 flags.go:64] FLAG: --manifest-url="" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500540 4441 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:12:50.508231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500545 4441 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500548 4441 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500552 4441 flags.go:64] FLAG: --max-pods="110" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500555 4441 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500558 4441 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500561 4441 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500564 4441 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500567 4441 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500570 4441 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500573 4441 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500579 4441 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500583 4441 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500586 4441 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500591 4441 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500594 4441 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500597 4441 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500600 4441 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500603 4441 flags.go:64] FLAG: --port="10250" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500606 4441 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500609 4441 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-070cd3cbfd8637ccc" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500612 4441 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500615 4441 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500618 4441 flags.go:64] FLAG: --register-node="true" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500621 4441 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500625 4441 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500628 4441 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:12:50.508910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500631 4441 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500634 4441 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500637 4441 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500641 4441 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500644 4441 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500647 4441 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500650 4441 flags.go:64] FLAG: --runonce="false" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500653 4441 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500656 4441 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500659 4441 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500662 4441 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500665 4441 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500668 4441 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500671 4441 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500674 4441 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500677 4441 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500680 4441 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500683 4441 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500686 4441 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500689 4441 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500693 4441 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500695 4441 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500700 4441 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500703 4441 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500706 4441 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:12:50.509616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500711 4441 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500713 4441 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500716 4441 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500719 4441 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500722 4441 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500725 4441 flags.go:64] FLAG: --v="2" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500729 4441 flags.go:64] FLAG: --version="false" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500733 4441 flags.go:64] FLAG: --vmodule="" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500737 4441 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.500740 4441 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500864 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500868 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500872 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500876 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500879 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500882 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500885 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500888 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500890 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500893 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500896 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500898 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.510289 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500901 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500922 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500925 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500928 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500933 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500937 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500941 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500945 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500949 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500954 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500957 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500960 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500963 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500976 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500979 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500982 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500985 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500987 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500990 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500993 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.511144 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500995 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.500998 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501001 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501003 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501006 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501009 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501011 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501015 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501017 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501020 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501022 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501025 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501028 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501030 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501033 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501035 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501038 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501041 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501044 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.511838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501046 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501049 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501051 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501054 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501057 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501059 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501062 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501064 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501067 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501069 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501072 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501075 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501077 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501080 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501082 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501085 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501087 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501090 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501093 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501095 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.512500 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501098 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501101 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501104 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501106 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501109 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501112 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501116 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501119 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501121 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501124 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501126 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501129 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501131 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501134 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501136 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501139 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501141 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501144 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501147 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.513005 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501149 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501152 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501154 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501157 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501159 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501162 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501164 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501167 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501169 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501172 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501176 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501179 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501182 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501185 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501187 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.513513 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501191 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.501217 4441 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501257 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501264 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501269 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501272 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501275 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501278 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501280 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501283 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501285 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501288 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501291 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501293 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501296 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501299 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.513912 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501301 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501304 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501306 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501309 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501311 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501314 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501317 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501320 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501322 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501325 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501327 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501330 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501332 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501335 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501337 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501340 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501342 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501345 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501348 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.514362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501353 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501357 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501360 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501362 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501365 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501367 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501370 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501373 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501375 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501378 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501380 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501383 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501386 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501388 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501391 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501393 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501396 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501398 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501401 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501403 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.514852 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501406 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501412 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501414 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501417 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501420 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501423 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501425 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501427 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501430 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501432 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501435 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501438 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501441 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501445 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501448 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501451 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501453 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501456 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501458 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501461 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.515423 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501463 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501466 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501468 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501471 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501473 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501476 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501479 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501481 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501484 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501486 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501489 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501493 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501496 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501499 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501502 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501505 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501507 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501510 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501512 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.515894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501516 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501520 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501523 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501526 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501528 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501538 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501542 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501545 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501548 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501550 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501553 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501556 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501558 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.501561 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.506107 4441 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.506123 4441 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.506142 4441 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.506146 4441 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506201 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506206 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.516412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506210 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506213 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506216 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506219 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506221 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506224 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506227 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506229 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506232 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506235 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506237 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506242 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506245 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506249 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506252 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506256 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506259 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506262 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506264 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.516894 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506267 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506270 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506272 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506275 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506278 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506280 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506283 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506286 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506289 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506291 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506294 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506297 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506299 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506302 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506304 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506307 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506309 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506312 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506315 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.517374 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506317 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506320 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506322 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506325 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506328 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506331 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506333 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506336 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506339 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506342 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506345 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506348 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506350 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506352 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506355 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506358 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506361 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506363 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506366 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.517813 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506368 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506371 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506374 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506376 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506379 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506381 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506384 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506387 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506390 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506392 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506395 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506397 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506400 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506402 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506405 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506408 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506411 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506414 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506417 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506420 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.518408 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506423 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506426 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506428 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506431 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506433 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506437 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506440 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506444 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506447 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506449 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506452 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506454 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506457 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506459 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506462 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506465 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506467 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506470 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506473 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506475 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.519051 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506478 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506481 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506484 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506486 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506489 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506492 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506494 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.506520 4441 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506650 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506654 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506658 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506660 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506663 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506667 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506669 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506672 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.519744 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506676 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506680 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506684 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506690 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506695 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506700 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506703 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506715 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506718 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506720 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506723 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506726 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506729 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506731 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506734 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506736 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506739 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506742 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506744 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.520334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506747 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506750 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506752 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506755 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506758 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506760 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506763 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506765 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506768 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506771 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506773 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506777 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506780 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506782 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506784 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506787 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506790 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506793 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506795 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506798 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.520989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506800 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506803 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506806 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506808 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506811 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506813 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506816 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506820 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506824 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506827 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506830 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506832 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506835 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506838 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506840 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506843 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506845 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506848 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506850 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506853 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.521773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506855 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506858 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506860 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506863 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506865 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506868 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506871 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506873 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506876 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506878 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506881 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506884 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506886 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506889 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506891 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506893 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506896 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506898 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506901 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506903 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.522490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506906 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506908 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506911 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506913 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506916 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506919 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506921 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506923 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506926 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506928 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506931 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506933 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506936 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506938 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506941 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506943 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506946 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506949 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.523027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.506951 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.506989 4441 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507041 4441 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507045 4441 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507049 4441 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507052 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507056 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507058 4441 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507061 4441 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507064 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507066 4441 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507070 4441 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507072 4441 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507075 4441 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507078 4441 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507080 4441 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:50.523484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507083 4441 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507085 4441 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507088 4441 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507091 4441 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507093 4441 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507096 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507099 4441 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507102 4441 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507105 4441 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507109 4441 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507113 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507116 4441 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507119 4441 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507122 4441 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507125 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507127 4441 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507130 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507133 4441 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507135 4441 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507138 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:50.523952 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507140 4441 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507143 4441 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507145 4441 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507148 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507150 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507153 4441 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507155 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507158 4441 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507161 4441 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507163 4441 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507166 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507168 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507171 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507173 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507176 4441 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507178 4441 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507181 4441 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507183 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507186 4441 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:50.524620 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507189 4441 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507191 4441 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507194 4441 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507197 4441 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507199 4441 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507202 4441 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507204 4441 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507207 4441 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507209 4441 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507212 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507215 4441 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507217 4441 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507220 4441 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507222 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507225 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507227 4441 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507230 4441 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507233 4441 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507235 4441 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:50.525102 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507238 4441 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507240 4441 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507243 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507246 4441 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507249 4441 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507251 4441 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507254 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507256 4441 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507259 4441 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507261 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507264 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507267 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507269 4441 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507272 4441 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507276 4441 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507279 4441 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507282 4441 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507284 4441 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507287 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:50.525558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507289 4441 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507292 4441 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507294 4441 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507297 4441 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507300 4441 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507303 4441 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507305 4441 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507308 4441 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507310 4441 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507313 4441 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507316 4441 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507319 4441 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507322 4441 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507324 4441 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:50.507327 4441 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.507384 4441 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.508294 4441 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.508351 4441 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 21:12:50.526120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.508516 4441 server.go:1014] "Starting client certificate rotation" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.508658 4441 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.508712 4441 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:07:10 +0000 UTC" deadline="2028-05-03 03:34:34.446569455 +0000 UTC" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.508739 4441 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13134h21m43.937832622s" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.508946 4441 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.509949 4441 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.511868 4441 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.513346 4441 log.go:25] "Validated CRI v1 image API" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.514055 4441 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:12:50.526558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.515464 4441 fs.go:135] Filesystem UUIDs: map[0d71475c-4762-40c2-a10d-1566d5a99b82:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a42eaacf-de30-455a-a963-001421f114e4:/dev/nvme0n1p3] Nov 02 21:12:50.526795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.515484 4441 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12cccc653695c2f8688e36fc1f8a0178b10b1ad07bffd7e40496e89b6c0ca281/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12cccc653695c2f8688e36fc1f8a0178b10b1ad07bffd7e40496e89b6c0ca281/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e268033f5eadeafbc34910c3704fc8874dc8998013ebeec5ced0f0bd7c4c9ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e268033f5eadeafbc34910c3704fc8874dc8998013ebeec5ced0f0bd7c4c9ae/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20490739847bd2cb2a505698ba4b350dc173ad5f97f98481fc509b0bfc990d70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20490739847bd2cb2a505698ba4b350dc173ad5f97f98481fc509b0bfc990d70/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36c403b809929d29073ecb136ae3ba48c650a713384a3d657483c1a503106645/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36c403b809929d29073ecb136ae3ba48c650a713384a3d657483c1a503106645/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/714882cb48529c99c446f53b0c6e8b684b9490fcd9f278f8874c5cec8a0f2836/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/714882cb48529c99c446f53b0c6e8b684b9490fcd9f278f8874c5cec8a0f2836/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d6b9296626564dcd6a62bed0d396655a974fbdc253a6cc14a82477644044d61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d6b9296626564dcd6a62bed0d396655a974fbdc253a6cc14a82477644044d61/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9945efe52ef2c589a75181da82dbe968ddd93c4b0d8005141c048a6762bf49f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9945efe52ef2c589a75181da82dbe968ddd93c4b0d8005141c048a6762bf49f9/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf657ea95a67c9ec705b9743bba8e0265b013d3362b47e45f4f5d93c6cfa39e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf657ea95a67c9ec705b9743bba8e0265b013d3362b47e45f4f5d93c6cfa39e6/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2d1063ca9a922327272befe2debd015f25babb0f4b4aaac8ae3afd1158acd50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2d1063ca9a922327272befe2debd015f25babb0f4b4aaac8ae3afd1158acd50/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d32188a08fc267b64ca9ababa519fa63b542a085c78735f30a5ca0d523fe886b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d32188a08fc267b64ca9ababa519fa63b542a085c78735f30a5ca0d523fe886b/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d638416e67a97861c343008ee706cf3264e309f90080c300ef9a7dece064ada2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d638416e67a97861c343008ee706cf3264e309f90080c300ef9a7dece064ada2/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e332a04295deadd7974c3e44e8eee36de9dccf7e4f8360cfc823b992011f1b1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e332a04295deadd7974c3e44e8eee36de9dccf7e4f8360cfc823b992011f1b1f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f5e5ce7-b1e0-46a6-8170-c379721e6e45/volumes/kubernetes.io~projected/kube-api-access-jz4b5:{mountpoint:/var/lib/kubelet/pods/1f5e5ce7-b1e0-46a6-8170-c379721e6e45/volumes/kubernetes.io~projected/kube-api-access-jz4b5 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f5e5ce7-b1e0-46a6-8170-c379721e6e45/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1f5e5ce7-b1e0-46a6-8170-c379721e6e45/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fc48ea1-7dbc-40fb-8db0-4591f92b9016/volumes/kubernetes.io~projected/kube-api-access-6nwnh:{mountpoint:/var/lib/kubelet/pods/2fc48ea1-7dbc-40fb-8db0-4591f92b9016/volumes/kubernetes.io~projected/kube-api-access-6nwnh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/319bc936-3891-49ed-81b9-a19a76825441/volumes/kubernetes.io~projected/kube-api-access-hncfz:{mountpoint:/var/lib/kubelet/pods/319bc936-3891-49ed-81b9-a19a76825441/volumes/kubernetes.io~projected/kube-api-access-hncfz major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~projected/kube-api-access-2d8x8:{mountpoint:/var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~projected/kube-api-access-2d8x8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/585fc755-2964-49cf-a948-e99760facfa2/volumes/kubernetes.io~projected/kube-api-access-7xb4t:{mountpoint:/var/lib/kubelet/pods/585fc755-2964-49cf-a948-e99760facfa2/volumes/kubernetes.io~projected/kube-api-access-7xb4t major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5/volumes/kubernetes.io~projected/kube-api-access-vgbm7:{mountpoint:/var/lib/kubelet/pods/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5/volumes/kubernetes.io~projected/kube-api-access-vgbm7 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volumes/kubernetes.io~projected/kube-api-access-xfg8h:{mountpoint:/var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volumes/kubernetes.io~projected/kube-api-access-xfg8h major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7f9e303-8cb2-497a-98dc-c80092327b2b/volumes/kubernetes.io~projected/kube-api-access-8m6ts:{mountpoint:/var/lib/kubelet/pods/b7f9e303-8cb2-497a-98dc-c80092327b2b/volumes/kubernetes.io~projected/kube-api-access-8m6ts major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b97770c2-375e-409d-9795-5cea674e8f2d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b97770c2-375e-409d-9795-5cea674e8f2d/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c59c2bf5-87b9-4d66-9b73-deae3d541ddd/volumes/kubernetes.io~projected/kube-api-access-rsxjm:{mountpoint:/var/lib/kubelet/pods/c59c2bf5-87b9-4d66-9b73-deae3d541ddd/volumes/kubernetes.io~projected/kube-api-access-rsxjm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4f97caf-1c7a-484a-8b2d-a885eb56d793/volumes/kubernetes.io~projected/kube-api-access-vdczn:{mountpoint:/var/lib/kubelet/pods/d4f97caf-1c7a-484a-8b2d-a885eb56d793/volumes/kubernetes.io~projected/kube-api-access-vdczn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2498872-da17-4526-8749-ce90d1f7fb7c/volumes/kubernetes.io~projected/kube-api-access-tmdps:{mountpoint:/var/lib/kubelet/pods/e2498872-da17-4526-8749-ce90d1f7fb7c/volumes/kubernetes.io~projected/kube-api-access-tmdps major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f35a604c-01b2-42fb-846e-a2d3f54045b3/volumes/kubernetes.io~projected/kube-api-access-4j77w:{mountpoint:/var/lib/kubelet/pods/f35a604c-01b2-42fb-846e-a2d3f54045b3/volumes/kubernetes.io~projected/kube-api-access-4j77w major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b7f820-cae3-42ef-a19e-9c75d9323cbc/volumes/kubernetes.io~projected/kube-api-access-fz26f:{mountpoint:/var/lib/kubelet/pods/f8b7f820-cae3-42ef-a19e-9c75d9323cbc/volumes/kubernetes.io~projected/kube-api-access-fz26f major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fca08020-5c83-4f30-8ab3-054ba4bc6171/volumes/kubernetes.io~projected/kube-api-access-5g4rr:{mountpoint:/var/lib/kubelet/pods/fca08020-5c83-4f30-8ab3-054ba4bc6171/volumes/kubernetes.io~projected/kube-api-access-5g4rr major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/893321a26ccdd7239137b6ce0414fa44a1499ec66c7b79090db2e9c3ed2f1ccc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/23333a5029b79179f81bfd958b3eed9f682b05316cc94939d6924e56506d0c5e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/db8f5cb52fffa63694459a490600a53449a71d13eabdcf9bfbb3e43a7f41df67/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/06f678e09eb6705cb1f7f4cc997a1df778e445e6953aa5e0a3e717a2ba125eca/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/290fa64857858d7a7e23a4c43744945a917096e3db0856c9ba8180898765df56/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/82ed12e1f913f376e2aa760bf22f00df1c13ca3f16f6d16b7d7602326af263c2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e06148094e791065e9efbdac8b4f8e88eb87ff7efa1c7e1d3fc6491c2549a027/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ca31fcffd37b4c4cbb02acd37f742e3f751629f26b0489833972c0e9efa3672e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f2e1466f13f1acf97b7d8eade8c88d663136db3ed8c87aa2ed42d30e0cf7ab14/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b739c834a82f012a67ea8e232abc74f79b065e8fe9ea317262934bed8eb24c87/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/d06805c3d5ba8e73b8733e955632bbe5e4ef84449d64598e64c23b652ff88663/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/cf5bbf81f85ba41556ae0994805f38aa3454233b33f449f11d3feef606439a0a/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a384baed7d4ac561e80f1dc7f6f23d791320bfc1016fe77346342f606c5954c7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/93c8edaddd1f7873d6a19ae6a7df17d6933f788364cc03012d04eeecbf8bf5c9/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/04d95352f2dffdad47b7bc78df5115f50248058655f12a4cbb8df5ce8018cb7e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bf102f749aa80657b209b5d8bb03f71911260a50c0f81d16815e6210f1cdcc35/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/96a3914b575bfbeec0f700204b09aec66f143e2d216125a13cb3fbba91c3d678/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/80ca2c20296d88d8a8456269f59163d12bc5298a762ecfc5c4fac7d8306ba365/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6cda1404e2b968ccd0d37253da65d25f236369d9bd8ccfc180030177af0a2d7d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fc36d3861aec3538e5771d3558e541363e0e77e43abf38e7612bb3bc9d9a3e2b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7bfafade513b460a76e28e79eb2bb14df29609e826f8ef9a6a04026515e0bcea/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/679b52526b4ac7685b7fc2f0bf12b44b5ef88e4ea5029dd2629cbfd116b9d937/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/9835b654fdd4d187c2861d3e1edb0b833434b517c1a829142b7cc2603e00d40b/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/3824bac81913d762138f4f721f771b0c9894fd4c3fe4bc56ee1eb55076e60752/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/a1ea9321a0a92fa3647d7c2628ae80f520bb38b105378939fb126a9739da0a0a/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/28c9ef9d8bc5bb05e0fa04e30b7144f4984273f29af37d8389966dd12970501a/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/2e2643330e0f9fc982172df5afca809f8a3002adca15ffcdc6db449cb0f21303/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/eede6c6a90db33e9f6d222e4d4b30362afc34ffd409afb045d655e26c3a18717/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/dd5e233570681b22e4f4fc7fc2141414874991b5b0a074c728a727c85731ff2d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6aac50a596a2a1585c9738128ef1fa9a59120784f437dccfbba7f673a7b2567b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/5ae995141d07f231921d989bca43a555d7750c5daeb7e66825c509688ecc081a/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/84e2a31c5192b3acc31078a4296019064bad7ae5b3fcf02ff87515060d298b5a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/3e2f9efc2255c861cd850232211f413b6185fcf3a5ed34e385d6c95a2767fc00/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/59635758b825d3d8ac362f0722600473a21ad6fd5e15f822db6d1b08a088fc9c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8de953150e3b22cf34a6d3b4549af616b50a2a6db6a561d64971aa4dadd6375b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/065e63d4b08f80065ab8c2283c38877811f51876ccd544f1dcbf59219b903ef5/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3deea7538ec6e3167cce3d04e914666ec1f83d6dffa614b291a86c5fd4ed45ae/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/92eb4ed0a2fcc97c3d0904b2600ed687b13a4abcb8dcf01ebde049e3a01c1280/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.521763 4441 manager.go:217] Machine: {Timestamp:2026-11-02 21:12:50.520847325 +0000 UTC m=+0.076475652 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095660 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec226bba871b6f59b234e35cbc0302de SystemUUID:ec226bba-871b-6f59-b234-e35cbc0302de BootID:766251f0-938d-44be-9821-07337660ceef Filesystems:[{Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c59c2bf5-87b9-4d66-9b73-deae3d541ddd/volumes/kubernetes.io~projected/kube-api-access-rsxjm DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e332a04295deadd7974c3e44e8eee36de9dccf7e4f8360cfc823b992011f1b1f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20490739847bd2cb2a505698ba4b350dc173ad5f97f98481fc509b0bfc990d70/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/319bc936-3891-49ed-81b9-a19a76825441/volumes/kubernetes.io~projected/kube-api-access-hncfz DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e2498872-da17-4526-8749-ce90d1f7fb7c/volumes/kubernetes.io~projected/kube-api-access-tmdps DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/714882cb48529c99c446f53b0c6e8b684b9490fcd9f278f8874c5cec8a0f2836/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf657ea95a67c9ec705b9743bba8e0265b013d3362b47e45f4f5d93c6cfa39e6/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b7f820-cae3-42ef-a19e-9c75d9323cbc/volumes/kubernetes.io~projected/kube-api-access-fz26f DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b97770c2-375e-409d-9795-5cea674e8f2d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fca08020-5c83-4f30-8ab3-054ba4bc6171/volumes/kubernetes.io~projected/kube-api-access-5g4rr DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2fc48ea1-7dbc-40fb-8db0-4591f92b9016/volumes/kubernetes.io~projected/kube-api-access-6nwnh DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f35a604c-01b2-42fb-846e-a2d3f54045b3/volumes/kubernetes.io~projected/kube-api-access-4j77w DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d4f97caf-1c7a-484a-8b2d-a885eb56d793/volumes/kubernetes.io~projected/kube-api-access-vdczn DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d6b9296626564dcd6a62bed0d396655a974fbdc253a6cc14a82477644044d61/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d638416e67a97861c343008ee706cf3264e309f90080c300ef9a7dece064ada2/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f5e5ce7-b1e0-46a6-8170-c379721e6e45/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~projected/kube-api-access-2d8x8 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/585fc755-2964-49cf-a948-e99760facfa2/volumes/kubernetes.io~projected/kube-api-access-7xb4t DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2d1063ca9a922327272befe2debd015f25babb0f4b4aaac8ae3afd1158acd50/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d32188a08fc267b64ca9ababa519fa63b542a085c78735f30a5ca0d523fe886b/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f5e5ce7-b1e0-46a6-8170-c379721e6e45/volumes/kubernetes.io~projected/kube-api-access-jz4b5 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3cba6681-dad1-4447-9c10-3eb9e98a7ebc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e268033f5eadeafbc34910c3704fc8874dc8998013ebeec5ced0f0bd7c4c9ae/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9945efe52ef2c589a75181da82dbe968ddd93c4b0d8005141c048a6762bf49f9/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5/volumes/kubernetes.io~projected/kube-api-access-vgbm7 DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b7f9e303-8cb2-497a-98dc-c80092327b2b/volumes/kubernetes.io~projected/kube-api-access-8m6ts DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8c954c9-200d-4823-b029-54cdebd356aa/volumes/kubernetes.io~projected/kube-api-access-xfg8h DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36c403b809929d29073ecb136ae3ba48c650a713384a3d657483c1a503106645/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12cccc653695c2f8688e36fc1f8a0178b10b1ad07bffd7e40496e89b6c0ca281/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:20490739847bd2c MacAddress:1e:98:38:9e:68:68 Speed:10000 Mtu:8901} {Name:714882cb48529c9 MacAddress:6a:33:62:02:8d:47 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fb:3c:06:75 Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:5e:a5:b6:20:34 Speed:0 Mtu:8901} {Name:cf657ea95a67c9e MacAddress:12:84:39:dc:b4:83 Speed:10000 Mtu:8901} {Name:d32188a08fc267b MacAddress:ae:0b:0b:2c:31:06 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fb:3c:06:75 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:56:7f:9d:fa:40:c3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:51:a6:31:e1:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522487 4441 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522550 4441 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522790 4441 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522809 4441 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-225.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522923 4441 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522931 4441 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522940 4441 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.522952 4441 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523045 4441 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523209 4441 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523308 4441 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523339 4441 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523348 4441 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523357 4441 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523365 4441 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523375 4441 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523911 4441 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.523920 4441 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:50.526934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.524392 4441 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:12:50.530612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.530598 4441 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:12:50.530612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.530611 4441 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:50.530711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.530701 4441 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 21:12:50.530883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.530872 4441 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:12:50.530915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.530901 4441 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:12:50.531060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531024 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:12:50.531060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531040 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:12:50.531060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531046 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:12:50.531060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531052 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:12:50.531060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531058 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531064 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531070 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531077 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531084 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531090 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531104 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531113 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531136 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:12:50.531191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.531142 4441 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:12:50.534349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.534334 4441 server.go:1263] "Started kubelet" Nov 02 21:12:50.534452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.534399 4441 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:12:50.534510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.534441 4441 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:12:50.534682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.534583 4441 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:12:50.534959 ip-10-0-137-225 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:12:50.535158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.535003 4441 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:12:50.536048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.535659 4441 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:12:50.536911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.536894 4441 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:12:50.537052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.537036 4441 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:12:50.537157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.537117 4441 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:07:12 +0000 UTC" deadline="2028-05-23 22:07:54.194340117 +0000 UTC" Nov 02 21:12:50.537157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.537148 4441 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13632h55m3.657195155s" Nov 02 21:12:50.537290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.537162 4441 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:12:50.537290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.537172 4441 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:12:50.537290 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.537196 4441 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:50.537290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.537245 4441 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539028 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8c954c9-200d-4823-b029-54cdebd356aa" volumeName="kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-env-overrides" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539095 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585fc755-2964-49cf-a948-e99760facfa2" volumeName="kubernetes.io/empty-dir/585fc755-2964-49cf-a948-e99760facfa2-data-volume" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539119 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5" volumeName="kubernetes.io/configmap/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-config-volume" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539134 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8c954c9-200d-4823-b029-54cdebd356aa" volumeName="kubernetes.io/projected/a8c954c9-200d-4823-b029-54cdebd356aa-kube-api-access-xfg8h" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539154 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8c954c9-200d-4823-b029-54cdebd356aa" volumeName="kubernetes.io/secret/a8c954c9-200d-4823-b029-54cdebd356aa-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539166 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fca08020-5c83-4f30-8ab3-054ba4bc6171" volumeName="kubernetes.io/configmap/fca08020-5c83-4f30-8ab3-054ba4bc6171-iptables-alerter-script" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539199 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5e5ce7-b1e0-46a6-8170-c379721e6e45" volumeName="kubernetes.io/secret/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-cert" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539217 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8c954c9-200d-4823-b029-54cdebd356aa" volumeName="kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-script-lib" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539233 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b97770c2-375e-409d-9795-5cea674e8f2d" volumeName="kubernetes.io/configmap/b97770c2-375e-409d-9795-5cea674e8f2d-konnectivity-ca" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539251 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fca08020-5c83-4f30-8ab3-054ba4bc6171" volumeName="kubernetes.io/projected/fca08020-5c83-4f30-8ab3-054ba4bc6171-kube-api-access-5g4rr" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539336 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fc48ea1-7dbc-40fb-8db0-4591f92b9016" volumeName="kubernetes.io/projected/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-kube-api-access-6nwnh" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539355 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585fc755-2964-49cf-a948-e99760facfa2" volumeName="kubernetes.io/projected/585fc755-2964-49cf-a948-e99760facfa2-kube-api-access-7xb4t" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.539384 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f97caf-1c7a-484a-8b2d-a885eb56d793" volumeName="kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-binary-copy" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540051 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35a604c-01b2-42fb-846e-a2d3f54045b3" volumeName="kubernetes.io/projected/f35a604c-01b2-42fb-846e-a2d3f54045b3-kube-api-access-4j77w" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540115 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cba6681-dad1-4447-9c10-3eb9e98a7ebc" volumeName="kubernetes.io/projected/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-kube-api-access-2d8x8" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540131 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5" volumeName="kubernetes.io/empty-dir/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-tmp-dir" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540152 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2498872-da17-4526-8749-ce90d1f7fb7c" volumeName="kubernetes.io/projected/e2498872-da17-4526-8749-ce90d1f7fb7c-kube-api-access-tmdps" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540166 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35a604c-01b2-42fb-846e-a2d3f54045b3" volumeName="kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-cni-binary-copy" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540183 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f97caf-1c7a-484a-8b2d-a885eb56d793" volumeName="kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540196 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f97caf-1c7a-484a-8b2d-a885eb56d793" volumeName="kubernetes.io/projected/d4f97caf-1c7a-484a-8b2d-a885eb56d793-kube-api-access-vdczn" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540215 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cba6681-dad1-4447-9c10-3eb9e98a7ebc" volumeName="kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-tuned" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540228 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cba6681-dad1-4447-9c10-3eb9e98a7ebc" volumeName="kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-tmp" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540251 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7f9e303-8cb2-497a-98dc-c80092327b2b" volumeName="kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540268 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="319bc936-3891-49ed-81b9-a19a76825441" volumeName="kubernetes.io/projected/319bc936-3891-49ed-81b9-a19a76825441-kube-api-access-hncfz" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540281 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c59c2bf5-87b9-4d66-9b73-deae3d541ddd" volumeName="kubernetes.io/projected/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kube-api-access-rsxjm" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540296 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35a604c-01b2-42fb-846e-a2d3f54045b3" volumeName="kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-daemon-config" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540330 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b7f820-cae3-42ef-a19e-9c75d9323cbc" volumeName="kubernetes.io/empty-dir/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-tmp-dir" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540342 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5" volumeName="kubernetes.io/projected/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-kube-api-access-vgbm7" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540368 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8c954c9-200d-4823-b029-54cdebd356aa" volumeName="kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-config" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540384 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b97770c2-375e-409d-9795-5cea674e8f2d" volumeName="kubernetes.io/secret/b97770c2-375e-409d-9795-5cea674e8f2d-agent-certs" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540397 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b7f820-cae3-42ef-a19e-9c75d9323cbc" volumeName="kubernetes.io/projected/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-kube-api-access-fz26f" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540409 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5e5ce7-b1e0-46a6-8170-c379721e6e45" volumeName="kubernetes.io/projected/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-kube-api-access-jz4b5" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540426 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f97caf-1c7a-484a-8b2d-a885eb56d793" volumeName="kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540438 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf2c02b-9cee-4a8b-a809-d2b52ec9235b" volumeName="kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540455 4441 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2498872-da17-4526-8749-ce90d1f7fb7c" volumeName="kubernetes.io/configmap/e2498872-da17-4526-8749-ce90d1f7fb7c-serviceca" seLinuxMountContext="" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540465 4441 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:12:50.541328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.540473 4441 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:12:50.543092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.542007 4441 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:12:50.543092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.542023 4441 factory.go:55] Registering systemd factory Nov 02 21:12:50.543092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.542039 4441 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:12:50.543092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.543024 4441 factory.go:153] Registering CRI-O factory Nov 02 21:12:50.543092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.543039 4441 factory.go:223] Registration of the crio container factory successfully Nov 02 21:12:50.543092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.543054 4441 factory.go:103] Registering Raw factory Nov 02 21:12:50.543092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.543066 4441 manager.go:1199] Started watching for new ooms in manager Nov 02 21:12:50.543996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.543960 4441 manager.go:319] Starting recovery of all containers Nov 02 21:12:50.549958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.549927 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:12:50.550090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.550073 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:12:50.574262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.574234 4441 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:12:50.575788 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.575767 4441 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:12:50.575788 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.575791 4441 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:12:50.575915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.575809 4441 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:12:50.575915 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.575844 4441 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 21:12:50.585481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.585460 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:12:50.585796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.585772 4441 generic.go:356] "Generic (PLEG): container finished" podID="a8c954c9-200d-4823-b029-54cdebd356aa" containerID="2820445b4802e0e96ef756754db20288c61f941605da1aaa8f8c386ee1aa852c" exitCode=1 Nov 02 21:12:50.599763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.599734 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="695e9a32f480399d7233e29900912bbc0996d58416e685f3900e0ab7eba31c31" exitCode=0 Nov 02 21:12:50.599831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.599767 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="fd07488fd7c9d9495a0558bcb08deef892188d358fd0d93f7112f3dc01bce5f5" exitCode=0 Nov 02 21:12:50.599831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.599782 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="7912e5d2372dfe1df92e6c829b2f9a86b5d4f717813ab1888a9e062a192e91c5" exitCode=0 Nov 02 21:12:50.599831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.599792 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="bd2a209e2d892a435a3733dbb80bad08ddbcbd11ed90ed3667e46791910d8229" exitCode=0 Nov 02 21:12:50.599831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.599803 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="dec4b87b24086362247941c6f9c6ba16ace51e11de291a73c3a85a62dd3039c5" exitCode=0 Nov 02 21:12:50.599831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.599810 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4f97caf-1c7a-484a-8b2d-a885eb56d793" containerID="f4dc3edbd8c7d164d67c1746df5bfb34939d0b5dba6abff863315a83c82309a3" exitCode=0 Nov 02 21:12:50.606460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.606434 4441 generic.go:356] "Generic (PLEG): container finished" podID="431b8dc08018bf9de47ad119a0b52d06" containerID="30e00b67f68a307c2eb6fe763882b3aaa780991e22033666bec44357b421363e" exitCode=0 Nov 02 21:12:50.606953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.606939 4441 manager.go:324] Recovery completed Nov 02 21:12:50.619646 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.619577 4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:50.620799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.620780 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:50.620914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.620814 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:50.620914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.620825 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:50.621997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.621984 4441 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:12:50.622048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.621998 4441 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:12:50.622048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622001 4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:50.622048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622015 4441 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:12:50.622191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622177 4441 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 21:12:50.622236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622192 4441 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 21:12:50.622236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622208 4441 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 21:12:50.622236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622214 4441 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 21:12:50.622236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622219 4441 policy_none.go:50] "Start" Nov 02 21:12:50.622236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622227 4441 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:12:50.622378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622238 4441 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:12:50.622412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622379 4441 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 21:12:50.622412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622388 4441 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 21:12:50.622412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622393 4441 policy_none.go:44] "Start" Nov 02 21:12:50.622808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622793 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:50.622848 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622821 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:50.622848 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.622834 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:50.626368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.626344 4441 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:12:50.629529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.629516 4441 manager.go:341] "Starting Device Plugin manager" Nov 02 21:12:50.629601 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.629546 4441 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:12:50.629601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.629554 4441 server.go:87] "Starting device plugin registration server" Nov 02 21:12:50.629828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.629816 4441 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:12:50.629880 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.629829 4441 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:12:50.629923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.629905 4441 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:12:50.630027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.629998 4441 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:12:50.630084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.630034 4441 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:12:50.630084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.630040 4441 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:12:50.630179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.630090 4441 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:12:50.630024529Z","UUID":"8c08b24f-c811-4165-98d1-cdd5d7107aa1","Handler":null,"Name":"","Endpoint":""} Nov 02 21:12:50.631287 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.631267 4441 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:12:50.631372 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.631310 4441 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" Nov 02 21:12:50.631372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.631330 4441 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:12:50.631372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.631351 4441 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:12:50.675412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.675393 4441 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:50.676429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.676393 4441 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal"] Nov 02 21:12:50.676517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.676498 4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:50.680243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.680228 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:50.680308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.680253 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:50.680308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.680263 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:50.680474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.680465 4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:50.680621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.680607 4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:50.681236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681220 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:50.681236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681242 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:50.681354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681252 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:50.681354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681292 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:50.681354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681312 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:50.681354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681328 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:50.681467 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.681375 4441 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.681522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681498 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"1b27bf1c010fcac7e9afcd7fd0096491","Type":"ContainerStarted","Data":"4c6017d89730c7f3dadfb8e5ba00f009eef5ab8694da8cbbb0913fd9f8b47363"} Nov 02 21:12:50.681560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681529 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"1b27bf1c010fcac7e9afcd7fd0096491","Type":"ContainerStarted","Data":"557feef4bce70e3354c78e5953ec33d37610a61b2881db91bb4269a5888d5d55"} Nov 02 21:12:50.681600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681590 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"431b8dc08018bf9de47ad119a0b52d06","Type":"ContainerStarted","Data":"4eaa27b6e7a9a6bea2cd2162cca66762ddd1098c61626cd5c87f05fe9b2d47f3"} Nov 02 21:12:50.681640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681604 4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:50.681686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681605 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"431b8dc08018bf9de47ad119a0b52d06","Type":"ContainerDied","Data":"30e00b67f68a307c2eb6fe763882b3aaa780991e22033666bec44357b421363e"} Nov 02 21:12:50.681728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.681712 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" event={"ID":"431b8dc08018bf9de47ad119a0b52d06","Type":"ContainerStarted","Data":"9d29f401dd06b7448836c645663d9633316bbd652aa021f63703bbcdbcfdea65"} Nov 02 21:12:50.682367 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.682353 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:50.682439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.682376 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:50.682439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.682385 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:50.682507 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.682492 4441 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-225.us-west-2.compute.internal\" not found" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.721982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.721947 4441 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:50.722116 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.722101 4441 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:50.730695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.730677 4441 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:50.731669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.731655 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:50.731752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.731683 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:50.731752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.731697 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:50.731752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.731719 4441 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.738902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.738888 4441 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.770792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.770775 4441 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 21:12:50.770882 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.770802 4441 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:50.816308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.816288 4441 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:50.816371 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.816326 4441 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.816371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.816338 4441 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.835157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.835140 4441 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.835214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.835194 4441 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.835633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.835619 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeNotReady" Nov 02 21:12:50.835689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.835650 4441 setters.go:546] "Node became not ready" node="ip-10-0-137-225.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T21:12:50Z","lastTransitionTime":"2026-11-02T21:12:50Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 21:12:50.871030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.870935 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.871030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.870961 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.871030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.870995 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b27bf1c010fcac7e9afcd7fd0096491-config\") pod \"kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"1b27bf1c010fcac7e9afcd7fd0096491\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.875265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.875249 4441 nodeinfomanager.go:471] existing CSINode "ip-10-0-137-225.us-west-2.compute.internal" is owned by different node (oldNodeID="50c7e98f-f037-46d9-970a-94e7248b21a8", newNodeID=""), cleaning up... Nov 02 21:12:50.899039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.899018 4441 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:50.899105 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:50.899059 4441 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.971663 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.971641 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.971746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.971671 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.971746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.971696 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b27bf1c010fcac7e9afcd7fd0096491-config\") pod \"kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"1b27bf1c010fcac7e9afcd7fd0096491\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.971746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.971741 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.971879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.971743 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431b8dc08018bf9de47ad119a0b52d06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"431b8dc08018bf9de47ad119a0b52d06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:50.971879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:50.971792 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b27bf1c010fcac7e9afcd7fd0096491-config\") pod \"kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal\" (UID: \"1b27bf1c010fcac7e9afcd7fd0096491\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:51.098514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.098494 4441 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-225.us-west-2.compute.internal" event="NodeReady" Nov 02 21:12:51.098615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.098558 4441 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:12:51.523783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.523757 4441 apiserver.go:52] "Watching apiserver" Nov 02 21:12:51.610792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.610770 4441 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:51.622543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.622522 4441 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:12:51.622953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.622934 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s95mt","openshift-ovn-kubernetes/ovnkube-node-wgl8r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal","openshift-image-registry/node-ca-4cd9w","openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9","openshift-network-operator/iptables-alerter-znl7n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5","kube-system/global-pull-secret-syncer-lsl4s","openshift-ingress-canary/ingress-canary-9cc7j","openshift-multus/multus-s75ln","kube-system/konnectivity-agent-h6889","openshift-cluster-node-tuning-operator/tuned-8v27x","openshift-dns/node-resolver-w8jd6","openshift-insights/insights-runtime-extractor-v8947","openshift-multus/multus-additional-cni-plugins-5zccx","openshift-multus/network-metrics-daemon-2fcgs","openshift-network-diagnostics/network-check-target-j5vdc"] Nov 02 21:12:51.623928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.623907 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:51.628185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.628159 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.630799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.630750 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.638129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.638111 4441 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:12:51.666921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.666894 4441 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:51.667012 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.666936 4441 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-225.us-west-2.compute.internal" Nov 02 21:12:51.674218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.674193 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z2vtc\"" Nov 02 21:12:51.674218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.674193 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.674350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.674220 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.674350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.674277 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nvsgt\"" Nov 02 21:12:51.674350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.674301 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r4znh\"" Nov 02 21:12:51.675298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.675281 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:12:51.676350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676333 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-cnibin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.676413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676364 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-multus\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.676413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676380 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-lib-modules\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.676413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676394 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.676413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676409 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-slash\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.676538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676457 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-config\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.676538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676492 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-kubelet\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.676596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676572 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-modprobe-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.676596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676593 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.676661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676617 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-conf\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.676661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676634 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-tuned\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.676661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676649 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cnibin\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.676761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676670 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-hosts-file\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:51.676761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676727 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-systemd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.676761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676753 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-tuned\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.676861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676763 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.676861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676789 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-socket-dir-parent\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.676861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676821 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-k8s-cni-cncf-io\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.676861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676839 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysconfig\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.677046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676865 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-etc-selinux\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.677046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676903 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-netd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.677046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.676947 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-cni-binary-copy\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.677046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677000 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.677205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677048 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2498872-da17-4526-8749-ce90d1f7fb7c-serviceca\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:51.677205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677155 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-registration-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.677462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677210 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-device-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.677462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677244 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-log-socket\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.677462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677288 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-original-pull-secret\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.677462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677292 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbm7\" (UniqueName: \"kubernetes.io/projected/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-kube-api-access-vgbm7\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.677462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677441 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585fc755-2964-49cf-a948-e99760facfa2-crio-socket\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677471 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz26f\" (UniqueName: \"kubernetes.io/projected/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-kube-api-access-fz26f\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677486 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-hostroot\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677507 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677526 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677564 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-systemd-units\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677604 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-var-lib-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677630 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-kubernetes\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.677704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677699 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-host\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677733 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677775 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdczn\" (UniqueName: \"kubernetes.io/projected/d4f97caf-1c7a-484a-8b2d-a885eb56d793-kube-api-access-vdczn\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677825 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdps\" (UniqueName: \"kubernetes.io/projected/e2498872-da17-4526-8749-ce90d1f7fb7c-kube-api-access-tmdps\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677846 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677861 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-ovn\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677875 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-bin\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677891 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-script-lib\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677906 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.677932 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4b5\" (UniqueName: \"kubernetes.io/projected/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-kube-api-access-jz4b5\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:51.678041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678041 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678067 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-kubelet-config\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678096 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-netns\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678129 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585fc755-2964-49cf-a948-e99760facfa2-data-volume\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678146 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-etc-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678199 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678203 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585fc755-2964-49cf-a948-e99760facfa2-data-volume\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678225 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-var-lib-kubelet\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678242 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-tmp\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678288 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-tmp\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678296 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678313 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-node-log\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678338 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-sys\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678354 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fca08020-5c83-4f30-8ab3-054ba4bc6171-iptables-alerter-script\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:51.678421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678401 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97770c2-375e-409d-9795-5cea674e8f2d-agent-certs\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678476 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwnh\" (UniqueName: \"kubernetes.io/projected/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-kube-api-access-6nwnh\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678507 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678537 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678562 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-etc-kubernetes\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678587 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-systemd\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678636 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-tmp-dir\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678674 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678700 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2498872-da17-4526-8749-ce90d1f7fb7c-host\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678706 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-tmp-dir\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678756 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfg8h\" (UniqueName: \"kubernetes.io/projected/a8c954c9-200d-4823-b029-54cdebd356aa-kube-api-access-xfg8h\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678779 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-bin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678796 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4rr\" (UniqueName: \"kubernetes.io/projected/fca08020-5c83-4f30-8ab3-054ba4bc6171-kube-api-access-5g4rr\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678812 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-tmp-dir\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678827 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-sys-fs\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678856 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-kubelet\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678872 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-env-overrides\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678896 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c954c9-200d-4823-b029-54cdebd356aa-ovn-node-metrics-cert\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678935 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-daemon-config\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678983 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j77w\" (UniqueName: \"kubernetes.io/projected/f35a604c-01b2-42fb-846e-a2d3f54045b3-kube-api-access-4j77w\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.678988 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-tmp-dir\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679008 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fca08020-5c83-4f30-8ab3-054ba4bc6171-host-slash\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679033 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-config-volume\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.679050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679053 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679089 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncfz\" (UniqueName: \"kubernetes.io/projected/319bc936-3891-49ed-81b9-a19a76825441-kube-api-access-hncfz\") pod \"migrator-66df5d496b-t82b9\" (UID: \"319bc936-3891-49ed-81b9-a19a76825441\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679141 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-dbus\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679171 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-socket-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679202 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-system-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679248 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-netns\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679279 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-cert\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679325 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xb4t\" (UniqueName: \"kubernetes.io/projected/585fc755-2964-49cf-a948-e99760facfa2-kube-api-access-7xb4t\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679355 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-system-cni-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679398 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxjm\" (UniqueName: \"kubernetes.io/projected/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kube-api-access-rsxjm\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679413 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-conf-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679512 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-multus-certs\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679527 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-run\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679579 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8x8\" (UniqueName: \"kubernetes.io/projected/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-kube-api-access-2d8x8\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.679854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.679615 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97770c2-375e-409d-9795-5cea674e8f2d-konnectivity-ca\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:51.774568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.774515 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.774732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.774712 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:12:51.774827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.774810 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.775006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.774990 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:12:51.775545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775531 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.775609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775572 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.775892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775861 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.776037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775893 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-q4vfp\"" Nov 02 21:12:51.776037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775904 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:12:51.776037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775913 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775893 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:12:51.776037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775896 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.775960 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776076 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776132 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776181 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776221 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-crvzm\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776229 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tjjtb\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776225 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776261 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:12:51.776356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776339 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6fz8\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776392 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776489 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776497 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776512 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776514 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ps8jp\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776518 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776551 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776557 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776561 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776531 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mnft8\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776594 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776629 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776631 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776642 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dtndn\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776631 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776651 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jzn4w\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776650 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776663 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776673 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:12:51.776752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776760 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sdlzm\"" Nov 02 21:12:51.777316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776809 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dfd2\"" Nov 02 21:12:51.777316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776848 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:12:51.777316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776914 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhk5v\"" Nov 02 21:12:51.777316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.776992 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:12:51.778131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.778114 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.778177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.778152 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2498872-da17-4526-8749-ce90d1f7fb7c-serviceca\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:51.778177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.778114 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.779247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.779223 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c954c9-200d-4823-b029-54cdebd356aa-ovn-node-metrics-cert\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.779381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.779363 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-daemon-config\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.779418 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.779363 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fca08020-5c83-4f30-8ab3-054ba4bc6171-iptables-alerter-script\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:51.779489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.779473 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97770c2-375e-409d-9795-5cea674e8f2d-agent-certs\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:51.779523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.779497 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-cert\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:51.780251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780233 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:12:51.780312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780268 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zzckx\"" Nov 02 21:12:51.780312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780277 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:12:51.780388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780304 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97770c2-375e-409d-9795-5cea674e8f2d-konnectivity-ca\") pod \"konnectivity-agent-h6889\" (UID: \"b97770c2-375e-409d-9795-5cea674e8f2d\") " pod="kube-system/konnectivity-agent-h6889" Nov 02 21:12:51.780456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780429 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.780499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780470 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:12:51.780499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780481 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c5zxb\"" Nov 02 21:12:51.780586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780539 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:12:51.780586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780566 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2498872-da17-4526-8749-ce90d1f7fb7c-host\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:51.780671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780589 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-bin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.780671 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.780626 4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:51.780671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780636 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-bin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.780671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780626 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-sys-fs\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780668 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-sys-fs\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780666 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-kubelet\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780705 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2498872-da17-4526-8749-ce90d1f7fb7c-host\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780705 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fca08020-5c83-4f30-8ab3-054ba4bc6171-host-slash\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780628 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780731 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-kubelet\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.780757 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:52.280729701 +0000 UTC m=+1.836358037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780777 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fca08020-5c83-4f30-8ab3-054ba4bc6171-host-slash\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780782 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780815 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-dbus\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.780840 4441 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:51.780855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780843 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-socket-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780866 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-system-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.780889 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:52.280875992 +0000 UTC m=+1.836504312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : configmap "kube-rbac-proxy" not found Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780888 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-dbus\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780896 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-socket-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780903 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-netns\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780924 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-netns\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780931 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-system-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780937 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-system-cni-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780956 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-conf-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780989 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-multus-certs\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.780995 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-system-cni-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781005 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-run\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781021 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-cnibin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781040 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-multus\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781060 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-multus-certs\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781064 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-lib-modules\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781082 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-cnibin\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781082 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-run\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781097 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781101 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-cni-multus\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781111 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-conf-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-slash\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781140 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-kubelet\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781151 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781152 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-slash\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781155 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-modprobe-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781183 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-var-lib-kubelet\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781156 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-lib-modules\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781229 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781249 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-conf\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781266 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-modprobe-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781293 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cnibin\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781310 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-hosts-file\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781323 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-systemd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781334 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-conf\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.781316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781339 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781344 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysctl-d\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781364 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-socket-dir-parent\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781387 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-systemd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781384 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-hosts-file\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781390 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-k8s-cni-cncf-io\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781420 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781419 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-socket-dir-parent\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781366 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cnibin\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781430 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-host-run-k8s-cni-cncf-io\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781442 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysconfig\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781487 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-sysconfig\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781492 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-etc-selinux\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781511 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-netd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781528 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-registration-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781545 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-device-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781581 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-netd\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781594 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-etc-selinux\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781595 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-device-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781612 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-log-socket\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781632 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585fc755-2964-49cf-a948-e99760facfa2-crio-socket\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781633 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-registration-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781654 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-log-socket\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781657 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-hostroot\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781677 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781692 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-hostroot\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781701 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-systemd-units\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781723 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585fc755-2964-49cf-a948-e99760facfa2-crio-socket\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781767 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-var-lib-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781788 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-kubernetes\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.781797 4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781805 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-var-lib-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781802 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-host\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781825 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-systemd-units\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781827 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-kubernetes\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.781842 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:52.281829053 +0000 UTC m=+1.837457373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : secret "metrics-daemon-secret" not found Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781867 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-host\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781886 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781912 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-ovn\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781938 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-bin\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781979 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-ovn\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781982 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.781963 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782000 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-cni-bin\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782006 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-kubelet-config\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.782041 4441 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782065 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-netns\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782038 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-netns\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:51.782082 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:52.28206988 +0000 UTC m=+1.837698197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782082 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbf2c02b-9cee-4a8b-a809-d2b52ec9235b-kubelet-config\") pod \"global-pull-secret-syncer-lsl4s\" (UID: \"cbf2c02b-9cee-4a8b-a809-d2b52ec9235b\") " pod="kube-system/global-pull-secret-syncer-lsl4s" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782109 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-etc-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782141 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-etc-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782145 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-var-lib-kubelet\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782163 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-node-log\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782178 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-sys\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782167 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-multus-cni-dir\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782200 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782226 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-var-lib-kubelet\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782228 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-node-log\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782240 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-sys\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782243 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782262 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-etc-kubernetes\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782265 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782271 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8c954c9-200d-4823-b029-54cdebd356aa-run-openvswitch\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782279 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-systemd\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782315 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-etc-systemd\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.782708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782324 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35a604c-01b2-42fb-846e-a2d3f54045b3-etc-kubernetes\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782794 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwnh\" (UniqueName: \"kubernetes.io/projected/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-kube-api-access-6nwnh\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782803 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4rr\" (UniqueName: \"kubernetes.io/projected/fca08020-5c83-4f30-8ab3-054ba4bc6171-kube-api-access-5g4rr\") pod \"iptables-alerter-znl7n\" (UID: \"fca08020-5c83-4f30-8ab3-054ba4bc6171\") " pod="openshift-network-operator/iptables-alerter-znl7n" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.782863 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xb4t\" (UniqueName: \"kubernetes.io/projected/585fc755-2964-49cf-a948-e99760facfa2-kube-api-access-7xb4t\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783051 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbm7\" (UniqueName: \"kubernetes.io/projected/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-kube-api-access-vgbm7\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783108 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8x8\" (UniqueName: \"kubernetes.io/projected/3cba6681-dad1-4447-9c10-3eb9e98a7ebc-kube-api-access-2d8x8\") pod \"tuned-8v27x\" (UID: \"3cba6681-dad1-4447-9c10-3eb9e98a7ebc\") " pod="openshift-cluster-node-tuning-operator/tuned-8v27x" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783144 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz26f\" (UniqueName: \"kubernetes.io/projected/f8b7f820-cae3-42ef-a19e-9c75d9323cbc-kube-api-access-fz26f\") pod \"node-resolver-w8jd6\" (UID: \"f8b7f820-cae3-42ef-a19e-9c75d9323cbc\") " pod="openshift-dns/node-resolver-w8jd6" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783165 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6ts\" (UniqueName: \"kubernetes.io/projected/b7f9e303-8cb2-497a-98dc-c80092327b2b-kube-api-access-8m6ts\") pod \"network-check-target-j5vdc\" (UID: \"b7f9e303-8cb2-497a-98dc-c80092327b2b\") " pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783341 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdps\" (UniqueName: \"kubernetes.io/projected/e2498872-da17-4526-8749-ce90d1f7fb7c-kube-api-access-tmdps\") pod \"node-ca-4cd9w\" (UID: \"e2498872-da17-4526-8749-ce90d1f7fb7c\") " pod="openshift-image-registry/node-ca-4cd9w" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783406 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfg8h\" (UniqueName: \"kubernetes.io/projected/a8c954c9-200d-4823-b029-54cdebd356aa-kube-api-access-xfg8h\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783531 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4b5\" (UniqueName: \"kubernetes.io/projected/1f5e5ce7-b1e0-46a6-8170-c379721e6e45-kube-api-access-jz4b5\") pod \"ingress-canary-9cc7j\" (UID: \"1f5e5ce7-b1e0-46a6-8170-c379721e6e45\") " pod="openshift-ingress-canary/ingress-canary-9cc7j" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783605 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j77w\" (UniqueName: \"kubernetes.io/projected/f35a604c-01b2-42fb-846e-a2d3f54045b3-kube-api-access-4j77w\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783626 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdczn\" (UniqueName: \"kubernetes.io/projected/d4f97caf-1c7a-484a-8b2d-a885eb56d793-kube-api-access-vdczn\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783678 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncfz\" (UniqueName: \"kubernetes.io/projected/319bc936-3891-49ed-81b9-a19a76825441-kube-api-access-hncfz\") pod \"migrator-66df5d496b-t82b9\" (UID: \"319bc936-3891-49ed-81b9-a19a76825441\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" Nov 02 21:12:51.784823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.783736 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxjm\" (UniqueName: \"kubernetes.io/projected/c59c2bf5-87b9-4d66-9b73-deae3d541ddd-kube-api-access-rsxjm\") pod \"aws-ebs-csi-driver-node-82cz5\" (UID: \"c59c2bf5-87b9-4d66-9b73-deae3d541ddd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82cz5" Nov 02 21:12:51.786921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.786899 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-config\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.788049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.788035 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f35a604c-01b2-42fb-846e-a2d3f54045b3-cni-binary-copy\") pod \"multus-s75ln\" (UID: \"f35a604c-01b2-42fb-846e-a2d3f54045b3\") " pod="openshift-multus/multus-s75ln" Nov 02 21:12:51.788206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.788194 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-ovnkube-script-lib\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.789166 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.789150 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8c954c9-200d-4823-b029-54cdebd356aa-env-overrides\") pod \"ovnkube-node-wgl8r\" (UID: \"a8c954c9-200d-4823-b029-54cdebd356aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:51.789203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.789178 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4f97caf-1c7a-484a-8b2d-a885eb56d793-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zccx\" (UID: \"d4f97caf-1c7a-484a-8b2d-a885eb56d793\") " pod="openshift-multus/multus-additional-cni-plugins-5zccx" Nov 02 21:12:51.789306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.789289 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-config-volume\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:51.932879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.932859 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:12:51.948308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:51.948269 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:52.068672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:52.068644 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:52.241839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:52.241791 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:52.285944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:52.285917 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:52.286072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:52.285953 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:52.286072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:52.286013 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:52.286072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:52.286044 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:52.286072 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286061 4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:52.286257 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286120 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:53.286102986 +0000 UTC m=+2.841731316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:52.286257 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286124 4441 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:52.286257 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286154 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:53.2861463 +0000 UTC m=+2.841774625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:52.286257 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286160 4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:52.286257 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286201 4441 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:52.286257 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286212 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:53.286198063 +0000 UTC m=+2.841826382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : secret "metrics-daemon-secret" not found Nov 02 21:12:52.286257 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:52.286258 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:53.286246477 +0000 UTC m=+2.841874792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : configmap "kube-rbac-proxy" not found Nov 02 21:12:52.809355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:52.809325 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:53.291859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:53.291830 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:53.291996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:53.291864 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:53.291996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:53.291887 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:53.291996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:53.291907 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:53.292092 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:53.292010 4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:53.292092 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:53.292069 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.292055062 +0000 UTC m=+4.847683381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:53.292092 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:53.292010 4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:53.292183 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:53.292099 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.292093124 +0000 UTC m=+4.847721438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : secret "metrics-daemon-secret" not found Nov 02 21:12:53.292183 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:53.292013 4441 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:53.292183 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:53.292127 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.292121931 +0000 UTC m=+4.847750247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:53.292330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:53.292315 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585fc755-2964-49cf-a948-e99760facfa2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:53.616766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:53.616736 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t82b9" event={"ID":"319bc936-3891-49ed-81b9-a19a76825441","Type":"ContainerStarted","Data":"7b0875b92570038c60ff1d3e0a672eee5a9a03399858abe14b8b2839b06c4f98"} Nov 02 21:12:54.458339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:54.458314 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:55.303164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:55.303127 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:55.303164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:55.303169 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:55.303452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:55.303193 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:55.303452 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:55.303294 4441 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:55.303452 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:55.303347 4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:55.303452 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:55.303370 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls podName:8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:59.303353222 +0000 UTC m=+8.858981537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls") pod "dns-default-s95mt" (UID: "8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5") : secret "dns-default-metrics-tls" not found Nov 02 21:12:55.303452 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:55.303305 4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:55.303452 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:55.303417 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:59.303398161 +0000 UTC m=+8.859026480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:55.303452 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:55.303436 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:59.303424179 +0000 UTC m=+8.859052507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : secret "metrics-daemon-secret" not found Nov 02 21:12:55.897181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:55.897148 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:56.438444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:56.438416 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w8jd6_f8b7f820-cae3-42ef-a19e-9c75d9323cbc/dns-node-resolver/0.log" Nov 02 21:12:57.235016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:57.234987 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4cd9w_e2498872-da17-4526-8749-ce90d1f7fb7c/node-ca/0.log" Nov 02 21:12:58.035559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:58.035510 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9cc7j_1f5e5ce7-b1e0-46a6-8170-c379721e6e45/serve-healthcheck-canary/0.log" Nov 02 21:12:58.129416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:58.129388 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:58.409601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:58.409561 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:58.541358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:58.541327 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:58.635172 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:58.635137 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-t82b9_319bc936-3891-49ed-81b9-a19a76825441/migrator/0.log" Nov 02 21:12:58.663535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:58.663467 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:59.010667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.010598 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:12:59.010826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.010812 4441 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:12:59.013371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.013354 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:12:59.051137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.051109 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:59.330203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.330173 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:12:59.330389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.330253 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:12:59.330389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.330284 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:59.330389 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:59.330325 4441 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:59.330389 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:59.330380 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls podName:585fc755-2964-49cf-a948-e99760facfa2 nodeName:}" failed. No retries permitted until 2026-11-02 21:13:07.330366936 +0000 UTC m=+16.885995265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v8947" (UID: "585fc755-2964-49cf-a948-e99760facfa2") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:59.330607 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:59.330408 4441 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:59.330607 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:12:59.330504 4441 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs podName:2fc48ea1-7dbc-40fb-8db0-4591f92b9016 nodeName:}" failed. No retries permitted until 2026-11-02 21:13:07.330485305 +0000 UTC m=+16.886113625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs") pod "network-metrics-daemon-2fcgs" (UID: "2fc48ea1-7dbc-40fb-8db0-4591f92b9016") : secret "metrics-daemon-secret" not found Nov 02 21:12:59.330769 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.330748 4441 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:12:59.332808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.332789 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5-metrics-tls\") pod \"dns-default-s95mt\" (UID: \"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5\") " pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:59.429260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.429233 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s95mt" Nov 02 21:12:59.613736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.613667 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:59.758642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.758611 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:59.797877 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:12:59.797847 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s95mt"] Nov 02 21:12:59.804926 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:12:59.804901 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf99ecd_a9ca_4176_a5e0_af7c1d68efc5.slice/crio-8cb12b1f34ac84d0adf768793062d2c22c11074be4228e7fd1deac03cad23c3a WatchSource:0}: Error finding container 8cb12b1f34ac84d0adf768793062d2c22c11074be4228e7fd1deac03cad23c3a: Status 404 returned error can't find the container with id 8cb12b1f34ac84d0adf768793062d2c22c11074be4228e7fd1deac03cad23c3a Nov 02 21:13:00.199501 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:00.199466 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:00.403537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:00.403503 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:00.631878 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:00.631845 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s95mt" event={"ID":"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5","Type":"ContainerStarted","Data":"8cb12b1f34ac84d0adf768793062d2c22c11074be4228e7fd1deac03cad23c3a"} Nov 02 21:13:00.798806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:00.798772 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:00.997925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:00.997852 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:01.603442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:01.603414 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:02.203883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.203859 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:02.399061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.398988 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:02.642779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.642745 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s95mt" event={"ID":"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5","Type":"ContainerStarted","Data":"9a2c626e0d46c6977098dddb058f680e819216781f76bdd3bd3c708d6a638129"} Nov 02 21:13:02.642933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.642832 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s95mt" event={"ID":"8cf99ecd-a9ca-4176-a5e0-af7c1d68efc5","Type":"ContainerStarted","Data":"e5aa82c68447215321000d564878b8c6ae61d746227dd8c2d74672e9863b234b"} Nov 02 21:13:02.642933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.642897 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s95mt" Nov 02 21:13:02.648964 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.648937 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wgl8r" Nov 02 21:13:02.794894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.794819 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:02.996081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:02.996042 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:03.597579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:03.597533 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:04.029407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:04.029330 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:04.196706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:04.196671 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:04.395798 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:04.395771 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:04.514576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:04.514545 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:04.632488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:04.632457 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:04.995269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:04.995238 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:05.023956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:05.023934 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h6889" Nov 02 21:13:05.123842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:05.123813 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:05.598141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:05.598111 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:06.000284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:06.000204 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:06.398581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:06.398551 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:06.743825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:06.743729 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:06.873437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:06.873407 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:07.197333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.197303 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:07.385117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.385088 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:13:07.385245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.385134 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:13:07.387482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.387460 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585fc755-2964-49cf-a948-e99760facfa2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v8947\" (UID: \"585fc755-2964-49cf-a948-e99760facfa2\") " pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:13:07.387590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.387463 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc48ea1-7dbc-40fb-8db0-4591f92b9016-metrics-certs\") pod \"network-metrics-daemon-2fcgs\" (UID: \"2fc48ea1-7dbc-40fb-8db0-4591f92b9016\") " pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:13:07.524326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.524265 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fcgs" Nov 02 21:13:07.531437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.531419 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v8947" Nov 02 21:13:07.612840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.611537 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:07.889119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.889085 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2fcgs"] Nov 02 21:13:07.940495 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:07.940468 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc48ea1_7dbc_40fb_8db0_4591f92b9016.slice/crio-06393115fc4a2c6af3607d3c9b53519dd2a8c96cf24fa2d08f16f7b7148368b4 WatchSource:0}: Error finding container 06393115fc4a2c6af3607d3c9b53519dd2a8c96cf24fa2d08f16f7b7148368b4: Status 404 returned error can't find the container with id 06393115fc4a2c6af3607d3c9b53519dd2a8c96cf24fa2d08f16f7b7148368b4 Nov 02 21:13:07.945380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:07.945360 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v8947"] Nov 02 21:13:07.953355 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:07.953334 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585fc755_2964_49cf_a948_e99760facfa2.slice/crio-bf13d4bb66870831e736d732f4e61c182f075d23780667c6b00aeae741aed21d WatchSource:0}: Error finding container bf13d4bb66870831e736d732f4e61c182f075d23780667c6b00aeae741aed21d: Status 404 returned error can't find the container with id bf13d4bb66870831e736d732f4e61c182f075d23780667c6b00aeae741aed21d Nov 02 21:13:08.001250 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:08.001211 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:08.396901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:08.396862 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:08.593406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:08.593374 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:08.663141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:08.663069 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8947" event={"ID":"585fc755-2964-49cf-a948-e99760facfa2","Type":"ContainerStarted","Data":"ed3044b923f98768781bebf31efec6477fc96214893d48de541562af75b89ef3"} Nov 02 21:13:08.663141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:08.663120 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8947" event={"ID":"585fc755-2964-49cf-a948-e99760facfa2","Type":"ContainerStarted","Data":"bf13d4bb66870831e736d732f4e61c182f075d23780667c6b00aeae741aed21d"} Nov 02 21:13:08.664177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:08.664156 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fcgs" event={"ID":"2fc48ea1-7dbc-40fb-8db0-4591f92b9016","Type":"ContainerStarted","Data":"06393115fc4a2c6af3607d3c9b53519dd2a8c96cf24fa2d08f16f7b7148368b4"} Nov 02 21:13:08.793695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:08.793667 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:09.669454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:09.669428 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8947" event={"ID":"585fc755-2964-49cf-a948-e99760facfa2","Type":"ContainerStarted","Data":"c4af46d2e7a672e4d454ba01b8e8929e8f282927f6534790a17eba02939c9e6d"} Nov 02 21:13:10.591430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.591394 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b76dc6b84-ftxt5"] Nov 02 21:13:10.593690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.593669 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.676778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.676737 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fcgs" event={"ID":"2fc48ea1-7dbc-40fb-8db0-4591f92b9016","Type":"ContainerStarted","Data":"4ccbe68a8c6b09dece033c0dbd5c5dd2a823ed4b5e41dc9d82a92c9c948204b6"} Nov 02 21:13:10.677287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.676799 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fcgs" event={"ID":"2fc48ea1-7dbc-40fb-8db0-4591f92b9016","Type":"ContainerStarted","Data":"8089f1997122c38e81329d660cf0ea14b813dd27d2177a5c5cbe2a6c6d9a21e2"} Nov 02 21:13:10.710494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710455 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32b5e36f-dc9e-4376-9431-eee805271cf1-image-registry-private-configuration\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.710642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710567 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-registry-tls\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.710642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710603 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32b5e36f-dc9e-4376-9431-eee805271cf1-registry-certificates\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.710642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710634 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32b5e36f-dc9e-4376-9431-eee805271cf1-installation-pull-secrets\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.710861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710651 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgkp\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-kube-api-access-4jgkp\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.710861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710676 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32b5e36f-dc9e-4376-9431-eee805271cf1-trusted-ca\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.710861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710832 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32b5e36f-dc9e-4376-9431-eee805271cf1-ca-trust-extracted\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.710861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.710861 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-bound-sa-token\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.739792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.739758 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 21:13:10.740146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.740124 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 21:13:10.740146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.740122 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 21:13:10.742799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.742779 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b8fk2\"" Nov 02 21:13:10.775463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.775440 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:10.778802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.778783 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 21:13:10.779214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.779193 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b76dc6b84-ftxt5"] Nov 02 21:13:10.811580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811553 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32b5e36f-dc9e-4376-9431-eee805271cf1-image-registry-private-configuration\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.811580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811581 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-registry-tls\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.811745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811605 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32b5e36f-dc9e-4376-9431-eee805271cf1-registry-certificates\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.811745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811684 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32b5e36f-dc9e-4376-9431-eee805271cf1-installation-pull-secrets\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.811745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811720 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgkp\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-kube-api-access-4jgkp\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.811745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811743 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32b5e36f-dc9e-4376-9431-eee805271cf1-trusted-ca\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.811922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811781 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32b5e36f-dc9e-4376-9431-eee805271cf1-ca-trust-extracted\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.811922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.811833 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-bound-sa-token\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.812409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.812379 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32b5e36f-dc9e-4376-9431-eee805271cf1-ca-trust-extracted\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.812553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.812534 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32b5e36f-dc9e-4376-9431-eee805271cf1-registry-certificates\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.812698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.812675 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32b5e36f-dc9e-4376-9431-eee805271cf1-trusted-ca\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.814721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.814699 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-registry-tls\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.814824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.814697 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32b5e36f-dc9e-4376-9431-eee805271cf1-installation-pull-secrets\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.814911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.814890 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32b5e36f-dc9e-4376-9431-eee805271cf1-image-registry-private-configuration\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.868020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.866151 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-bound-sa-token\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.868020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.866384 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgkp\" (UniqueName: \"kubernetes.io/projected/32b5e36f-dc9e-4376-9431-eee805271cf1-kube-api-access-4jgkp\") pod \"image-registry-5b76dc6b84-ftxt5\" (UID: \"32b5e36f-dc9e-4376-9431-eee805271cf1\") " pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:10.895527 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.895476 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:10.904728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:10.904703 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:11.170611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.170542 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:11.290669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.290632 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:11.325467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.325447 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b76dc6b84-ftxt5"] Nov 02 21:13:11.333278 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:11.333253 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b5e36f_dc9e_4376_9431_eee805271cf1.slice/crio-3075bc0af76aba68fa2ea83d8d5657d5703eb6d4c25a89c76f302664cb1ca374 WatchSource:0}: Error finding container 3075bc0af76aba68fa2ea83d8d5657d5703eb6d4c25a89c76f302664cb1ca374: Status 404 returned error can't find the container with id 3075bc0af76aba68fa2ea83d8d5657d5703eb6d4c25a89c76f302664cb1ca374 Nov 02 21:13:11.681713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.681680 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" event={"ID":"32b5e36f-dc9e-4376-9431-eee805271cf1","Type":"ContainerStarted","Data":"e350a07d010b2a3d6670bbe297eb776586c5fa5ae0b3021f0511285ac490272d"} Nov 02 21:13:11.682131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.681755 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" event={"ID":"32b5e36f-dc9e-4376-9431-eee805271cf1","Type":"ContainerStarted","Data":"3075bc0af76aba68fa2ea83d8d5657d5703eb6d4c25a89c76f302664cb1ca374"} Nov 02 21:13:11.682131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.681806 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:11.683733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.683708 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v8947" event={"ID":"585fc755-2964-49cf-a948-e99760facfa2","Type":"ContainerStarted","Data":"6b438398cb02a26ad7b027a805817b162a9b54494e42c841974be4dfa79b011c"} Nov 02 21:13:11.793424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.793375 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" podStartSLOduration=1.7933631559999998 podStartE2EDuration="1.793363156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:11.792313651 +0000 UTC m=+21.347941988" watchObservedRunningTime="2026-11-02 21:13:11.793363156 +0000 UTC m=+21.348991487" Nov 02 21:13:11.829399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.829378 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:11.949964 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:11.949881 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:12.088376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.088347 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:12.212761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.212693 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:12.648235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.648210 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s95mt" Nov 02 21:13:12.840141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.840110 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:12.923204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.923149 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz"] Nov 02 21:13:12.925877 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.925863 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" Nov 02 21:13:12.977939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.977914 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 21:13:12.978094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.978080 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xxbxr\"" Nov 02 21:13:12.978636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:12.978615 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz"] Nov 02 21:13:13.008123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:13.008096 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:13.029251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:13.029226 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/716b4f26-9f07-47ef-837f-11d591f9feea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-thzlz\" (UID: \"716b4f26-9f07-47ef-837f-11d591f9feea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" Nov 02 21:13:13.129497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:13.129469 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/716b4f26-9f07-47ef-837f-11d591f9feea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-thzlz\" (UID: \"716b4f26-9f07-47ef-837f-11d591f9feea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" Nov 02 21:13:13.131855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:13.131836 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/716b4f26-9f07-47ef-837f-11d591f9feea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-thzlz\" (UID: \"716b4f26-9f07-47ef-837f-11d591f9feea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" Nov 02 21:13:13.236020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:13.235919 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" Nov 02 21:13:13.596239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:13.596209 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz"] Nov 02 21:13:13.604557 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:13.604527 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716b4f26_9f07_47ef_837f_11d591f9feea.slice/crio-4bc89fa6690a521c282377a47851529a50ce5ffe78878f387ddd43edb000cf95 WatchSource:0}: Error finding container 4bc89fa6690a521c282377a47851529a50ce5ffe78878f387ddd43edb000cf95: Status 404 returned error can't find the container with id 4bc89fa6690a521c282377a47851529a50ce5ffe78878f387ddd43edb000cf95 Nov 02 21:13:13.691224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:13.691196 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" event={"ID":"716b4f26-9f07-47ef-837f-11d591f9feea","Type":"ContainerStarted","Data":"4bc89fa6690a521c282377a47851529a50ce5ffe78878f387ddd43edb000cf95"} Nov 02 21:13:15.701241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:15.701206 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" event={"ID":"716b4f26-9f07-47ef-837f-11d591f9feea","Type":"ContainerStarted","Data":"eb1862c679220e2625dea198db7325bad2aeb7ff9f88b1df20ee960f01f338e2"} Nov 02 21:13:15.701610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:15.701300 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" Nov 02 21:13:15.705837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:15.705816 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" Nov 02 21:13:15.808047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:15.807998 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-thzlz" podStartSLOduration=1.851317405 podStartE2EDuration="3.807984216s" totalImagesPullingTime="1.956666811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:15.806898515 +0000 UTC m=+25.362526851" watchObservedRunningTime="2026-11-02 21:13:15.807984216 +0000 UTC m=+25.363612544" Nov 02 21:13:16.101759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.101729 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:16.224810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.224777 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:16.339677 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.339647 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:16.459955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.459885 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:16.556589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.556563 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vb59j"] Nov 02 21:13:16.560632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.560616 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.650440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.650413 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/843684e3-80f3-46fe-9441-fc2b5abaf6dd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.650557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.650459 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/843684e3-80f3-46fe-9441-fc2b5abaf6dd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.650557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.650538 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/843684e3-80f3-46fe-9441-fc2b5abaf6dd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.650637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.650578 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfg5x\" (UniqueName: \"kubernetes.io/projected/843684e3-80f3-46fe-9441-fc2b5abaf6dd-kube-api-access-hfg5x\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.659875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.659849 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:13:16.660256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.660239 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 21:13:16.660921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.660904 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vb59j"] Nov 02 21:13:16.707438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.707420 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:13:16.707438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.707433 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 21:13:16.707782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.707420 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gsgb5\"" Nov 02 21:13:16.707782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.707480 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:13:16.751894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.751834 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/843684e3-80f3-46fe-9441-fc2b5abaf6dd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.751894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.751867 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/843684e3-80f3-46fe-9441-fc2b5abaf6dd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.752073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.751895 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/843684e3-80f3-46fe-9441-fc2b5abaf6dd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.752073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.751930 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfg5x\" (UniqueName: \"kubernetes.io/projected/843684e3-80f3-46fe-9441-fc2b5abaf6dd-kube-api-access-hfg5x\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.752536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.752508 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/843684e3-80f3-46fe-9441-fc2b5abaf6dd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.754222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.754197 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/843684e3-80f3-46fe-9441-fc2b5abaf6dd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.754295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.754243 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/843684e3-80f3-46fe-9441-fc2b5abaf6dd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.805833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.805809 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfg5x\" (UniqueName: \"kubernetes.io/projected/843684e3-80f3-46fe-9441-fc2b5abaf6dd-kube-api-access-hfg5x\") pod \"prometheus-operator-9cd89b74c-vb59j\" (UID: \"843684e3-80f3-46fe-9441-fc2b5abaf6dd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:16.873775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:16.873749 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" Nov 02 21:13:17.234711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:17.234673 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vb59j"] Nov 02 21:13:17.274612 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:17.274582 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843684e3_80f3_46fe_9441_fc2b5abaf6dd.slice/crio-a1bfff50185c34f438f5c4436e5f1762d4a118b7d1a90458a2844b617937af18 WatchSource:0}: Error finding container a1bfff50185c34f438f5c4436e5f1762d4a118b7d1a90458a2844b617937af18: Status 404 returned error can't find the container with id a1bfff50185c34f438f5c4436e5f1762d4a118b7d1a90458a2844b617937af18 Nov 02 21:13:17.707960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:17.707924 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" event={"ID":"843684e3-80f3-46fe-9441-fc2b5abaf6dd","Type":"ContainerStarted","Data":"a1bfff50185c34f438f5c4436e5f1762d4a118b7d1a90458a2844b617937af18"} Nov 02 21:13:19.016378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:19.016352 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j5vdc" Nov 02 21:13:19.716767 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:19.716731 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" event={"ID":"843684e3-80f3-46fe-9441-fc2b5abaf6dd","Type":"ContainerStarted","Data":"be4ae7dde03dd0696c095a0f27a03a9330fedb372c86b81ff56a63ce92f66a8c"} Nov 02 21:13:19.716963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:19.716798 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" event={"ID":"843684e3-80f3-46fe-9441-fc2b5abaf6dd","Type":"ContainerStarted","Data":"2b9dd4f1d4eeb4b0e054300d4b3dc24481cbe88d2896e120f668971c85ec309d"} Nov 02 21:13:19.829251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:19.829204 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vb59j" podStartSLOduration=2.083619077 podStartE2EDuration="3.829190391s" totalImagesPullingTime="1.745571314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:19.828390504 +0000 UTC m=+29.384018836" watchObservedRunningTime="2026-11-02 21:13:19.829190391 +0000 UTC m=+29.384818722" Nov 02 21:13:21.528622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:21.528571 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:21.661821 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:21.661785 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:21.784016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:21.783930 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:22.418576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:22.418542 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:22.606431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:22.606398 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:22.725882 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:22.725792 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:22.848205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:22.848175 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:22.997183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:22.997111 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:23.022554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.022527 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rs2wz"] Nov 02 21:13:23.057925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.057903 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.107793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.107769 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:13:23.107903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.107817 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:13:23.117840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.117815 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:23.154792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.154773 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:13:23.154849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.154833 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hbtfx\"" Nov 02 21:13:23.198196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198156 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198247 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-metrics-client-ca\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198280 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-tls\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198310 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-textfile\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198408 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdkg\" (UniqueName: \"kubernetes.io/projected/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-kube-api-access-qfdkg\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198429 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-root\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198444 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-wtmp\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198464 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-sys\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.198520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.198501 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299335 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-root\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299367 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-wtmp\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299386 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-sys\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299409 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299448 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-root\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299438 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299517 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-metrics-client-ca\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299544 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-tls\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299588 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-textfile\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.299773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.299635 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdkg\" (UniqueName: \"kubernetes.io/projected/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-kube-api-access-qfdkg\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.300097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.300074 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.300247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.300231 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-wtmp\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.300312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.300287 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-sys\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.300393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.300378 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-metrics-client-ca\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.300728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.300711 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-textfile\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.302621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.302596 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-tls\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.303193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.303175 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.346177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.346153 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:23.353462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.353442 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdkg\" (UniqueName: \"kubernetes.io/projected/24ae74e6-4a58-4e6e-9a7a-e8b33853846e-kube-api-access-qfdkg\") pod \"node-exporter-rs2wz\" (UID: \"24ae74e6-4a58-4e6e-9a7a-e8b33853846e\") " pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.406939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.406919 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rs2wz" Nov 02 21:13:23.435824 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:23.435804 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ae74e6_4a58_4e6e_9a7a_e8b33853846e.slice/crio-2f365701062b73064e075e173c175b26bbcb0115b4361d704fe19f5dacaab07e WatchSource:0}: Error finding container 2f365701062b73064e075e173c175b26bbcb0115b4361d704fe19f5dacaab07e: Status 404 returned error can't find the container with id 2f365701062b73064e075e173c175b26bbcb0115b4361d704fe19f5dacaab07e Nov 02 21:13:23.651791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.651765 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:23.730264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:23.730237 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs2wz" event={"ID":"24ae74e6-4a58-4e6e-9a7a-e8b33853846e","Type":"ContainerStarted","Data":"2f365701062b73064e075e173c175b26bbcb0115b4361d704fe19f5dacaab07e"} Nov 02 21:13:25.737715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:25.737677 4441 generic.go:356] "Generic (PLEG): container finished" podID="24ae74e6-4a58-4e6e-9a7a-e8b33853846e" containerID="76c53148eacf97dbe0b5271fd3069872657a25c4dd10ba17f05328b84d1603cb" exitCode=0 Nov 02 21:13:25.738129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:25.737731 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs2wz" event={"ID":"24ae74e6-4a58-4e6e-9a7a-e8b33853846e","Type":"ContainerDied","Data":"76c53148eacf97dbe0b5271fd3069872657a25c4dd10ba17f05328b84d1603cb"} Nov 02 21:13:26.300756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:26.300722 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:26.544398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:26.544368 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:26.743536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:26.743483 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs2wz" event={"ID":"24ae74e6-4a58-4e6e-9a7a-e8b33853846e","Type":"ContainerStarted","Data":"23801a3797dd22befa5402fb0af37eeae142f6df57698e4d69a83db5e4df96b3"} Nov 02 21:13:26.743901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:26.743563 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs2wz" event={"ID":"24ae74e6-4a58-4e6e-9a7a-e8b33853846e","Type":"ContainerStarted","Data":"b97dc62729d917a3c0a533c344765733816ad87ceece0bdbc7a892332f02af54"} Nov 02 21:13:26.852898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:26.852855 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rs2wz" podStartSLOduration=3.068315197 podStartE2EDuration="4.852843058s" totalImagesPullingTime="1.720155908s" totalInitContainerRuntime="64.371953ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:26.852183506 +0000 UTC m=+36.407811843" watchObservedRunningTime="2026-11-02 21:13:26.852843058 +0000 UTC m=+36.408471394" Nov 02 21:13:27.199290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:27.199249 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:27.735462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:27.735432 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:28.107836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:28.107793 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:29.001236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:29.001189 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:29.117854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:29.117817 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:29.237741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:29.237711 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:29.354305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:29.354276 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:30.299937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:30.299891 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:30.425912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:30.425877 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:30.540951 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:30.540920 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:30.660082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:30.660056 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:30.967067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:30.966996 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:31.084003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.083959 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:31.195281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.195250 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:31.363326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.363292 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:31.492987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.492940 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:31.613768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.613690 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7qzpl"] Nov 02 21:13:31.615820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.615798 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:31.665575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.665550 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 21:13:31.665658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.665586 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qn25f\"" Nov 02 21:13:31.665658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.665648 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 21:13:31.713376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.713353 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7qzpl"] Nov 02 21:13:31.759912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.759887 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhddd\" (UniqueName: \"kubernetes.io/projected/3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f-kube-api-access-dhddd\") pod \"downloads-6dd789c8b7-7qzpl\" (UID: \"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f\") " pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:31.760026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.759916 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f-tmp\") pod \"downloads-6dd789c8b7-7qzpl\" (UID: \"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f\") " pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:31.860171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.860145 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhddd\" (UniqueName: \"kubernetes.io/projected/3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f-kube-api-access-dhddd\") pod \"downloads-6dd789c8b7-7qzpl\" (UID: \"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f\") " pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:31.860262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.860177 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f-tmp\") pod \"downloads-6dd789c8b7-7qzpl\" (UID: \"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f\") " pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:31.860575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.860558 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f-tmp\") pod \"downloads-6dd789c8b7-7qzpl\" (UID: \"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f\") " pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:31.913465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.913411 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhddd\" (UniqueName: \"kubernetes.io/projected/3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f-kube-api-access-dhddd\") pod \"downloads-6dd789c8b7-7qzpl\" (UID: \"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f\") " pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:31.926112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:31.926053 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:32.389628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:32.389599 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7qzpl"] Nov 02 21:13:32.397485 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:32.397455 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb51bc8_4300_4b5e_a3b8_6f0118c13d4f.slice/crio-d79184cd7d35e551b0c971d492bbf64a515c3f6741daffc30ef613e05d0f99b8 WatchSource:0}: Error finding container d79184cd7d35e551b0c971d492bbf64a515c3f6741daffc30ef613e05d0f99b8: Status 404 returned error can't find the container with id d79184cd7d35e551b0c971d492bbf64a515c3f6741daffc30ef613e05d0f99b8 Nov 02 21:13:32.691797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:32.691735 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b76dc6b84-ftxt5" Nov 02 21:13:32.764419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:32.764379 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7qzpl" event={"ID":"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f","Type":"ContainerStarted","Data":"d79184cd7d35e551b0c971d492bbf64a515c3f6741daffc30ef613e05d0f99b8"} Nov 02 21:13:32.917379 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:32.917349 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:33.033136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:33.033058 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:33.330483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:33.330447 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:33.448775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:33.448729 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:34.008393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:34.008359 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:34.953990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:34.953948 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:35.073615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:35.073575 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:35.196850 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:35.196798 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:36.734238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:36.734187 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:36.903196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:36.903162 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:37.768958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:37.768921 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:37.885960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:37.885921 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:40.616482 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:13:40.616435 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:13:40.777561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:40.777528 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:40.967438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:40.967355 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:41.612981 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.612944 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85887bddc7-vgf66"] Nov 02 21:13:41.618836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.618814 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.626466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.626155 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv7zs\" (UniqueName: \"kubernetes.io/projected/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-kube-api-access-sv7zs\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.626466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.626294 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-oauth-config\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.626466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.626347 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-service-ca\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.626466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.626380 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-tmp\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.626466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.626406 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-oauth-serving-cert\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.626691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.626529 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-config\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.626691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.626579 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-serving-cert\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.718148 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.718092 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 21:13:41.718148 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.718113 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 21:13:41.718399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.718155 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 21:13:41.718399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.718158 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 21:13:41.718399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.718218 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 21:13:41.718528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.718398 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kdlnp\"" Nov 02 21:13:41.719318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.719290 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85887bddc7-vgf66"] Nov 02 21:13:41.726853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.726824 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv7zs\" (UniqueName: \"kubernetes.io/projected/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-kube-api-access-sv7zs\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.727001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.726875 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-oauth-config\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.727001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.726916 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-service-ca\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.727001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.726953 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-tmp\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.727176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.726999 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-oauth-serving-cert\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.727176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.727033 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-config\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.727176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.727117 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-serving-cert\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.728187 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.728159 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-tmp\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.728308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.728263 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-service-ca\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.728804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.728785 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-oauth-serving-cert\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.729323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.729304 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-config\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.730365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.730342 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-serving-cert\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.734569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.734551 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-oauth-config\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.792917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.792887 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv7zs\" (UniqueName: \"kubernetes.io/projected/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-kube-api-access-sv7zs\") pod \"console-85887bddc7-vgf66\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:41.876301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.876218 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:41.932040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:41.932002 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:42.077713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:42.077665 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:42.305073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:42.304994 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85887bddc7-vgf66"] Nov 02 21:13:42.344292 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:42.344258 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a12c79_6df1_4f1f_bea0_1c32dce168f2.slice/crio-11fa2d8108417ea6a0c866a6ccbb5cda9e7f45e5a0d114440cec8e949c65098f WatchSource:0}: Error finding container 11fa2d8108417ea6a0c866a6ccbb5cda9e7f45e5a0d114440cec8e949c65098f: Status 404 returned error can't find the container with id 11fa2d8108417ea6a0c866a6ccbb5cda9e7f45e5a0d114440cec8e949c65098f Nov 02 21:13:42.806749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:42.806716 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85887bddc7-vgf66" event={"ID":"f4a12c79-6df1-4f1f-bea0-1c32dce168f2","Type":"ContainerStarted","Data":"11fa2d8108417ea6a0c866a6ccbb5cda9e7f45e5a0d114440cec8e949c65098f"} Nov 02 21:13:46.825900 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:46.825862 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85887bddc7-vgf66" event={"ID":"f4a12c79-6df1-4f1f-bea0-1c32dce168f2","Type":"ContainerStarted","Data":"cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d"} Nov 02 21:13:46.935177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:46.935128 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85887bddc7-vgf66" podStartSLOduration=2.5231189880000002 podStartE2EDuration="5.935112763s" totalImagesPullingTime="3.411993775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:46.934417368 +0000 UTC m=+56.490045705" watchObservedRunningTime="2026-11-02 21:13:46.935112763 +0000 UTC m=+56.490741101" Nov 02 21:13:49.841641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:49.841599 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7qzpl" event={"ID":"3fb51bc8-4300-4b5e-a3b8-6f0118c13d4f","Type":"ContainerStarted","Data":"a4af6290bcb25075cb5b25834c7fc8681af0b8cf7c0f30330f1e8f6b878c9ff0"} Nov 02 21:13:49.842185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:49.841927 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:49.855837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:49.855808 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7qzpl" Nov 02 21:13:49.971516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:49.971485 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:50.103385 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.103312 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:50.110275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.110220 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7qzpl" podStartSLOduration=2.239023924 podStartE2EDuration="19.110200929s" totalImagesPullingTime="16.871177005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:50.043735826 +0000 UTC m=+59.599364163" watchObservedRunningTime="2026-11-02 21:13:50.110200929 +0000 UTC m=+59.665829326" Nov 02 21:13:50.110536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.110515 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c7c4469fd-r8dll"] Nov 02 21:13:50.115063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.115039 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.170751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.170594 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7c4469fd-r8dll"] Nov 02 21:13:50.197545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197511 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-oauth-config\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.197545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197552 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-config\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.197765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197715 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqch\" (UniqueName: \"kubernetes.io/projected/6e0f3822-e583-4d83-bfe7-809d70dd0824-kube-api-access-4zqch\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.197765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197749 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-service-ca\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.197861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197798 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-trusted-ca-bundle\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.197861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197839 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-serving-cert\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.197947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197862 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-oauth-serving-cert\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.197947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.197894 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e0f3822-e583-4d83-bfe7-809d70dd0824-tmp\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.250770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.250736 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 21:13:50.299091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299059 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqch\" (UniqueName: \"kubernetes.io/projected/6e0f3822-e583-4d83-bfe7-809d70dd0824-kube-api-access-4zqch\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.299269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299103 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-service-ca\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.299269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299138 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-trusted-ca-bundle\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.299269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299175 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-serving-cert\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.299269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299195 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-oauth-serving-cert\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.299269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299215 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e0f3822-e583-4d83-bfe7-809d70dd0824-tmp\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.299269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299269 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-oauth-config\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.299602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299299 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-config\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.300392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.299991 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-service-ca\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.300392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.300048 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e0f3822-e583-4d83-bfe7-809d70dd0824-tmp\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.300392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.300201 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-oauth-serving-cert\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.300392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.300268 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-trusted-ca-bundle\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.300719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.300697 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-config\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.302815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.302788 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-serving-cert\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.303140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.303111 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-oauth-config\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.353781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.353716 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqch\" (UniqueName: \"kubernetes.io/projected/6e0f3822-e583-4d83-bfe7-809d70dd0824-kube-api-access-4zqch\") pod \"console-7c7c4469fd-r8dll\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.428403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.428365 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:13:50.809030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.808998 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7c4469fd-r8dll"] Nov 02 21:13:50.817296 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:13:50.817257 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0f3822_e583_4d83_bfe7_809d70dd0824.slice/crio-877f384478fb31c291170b684bf6805a2500d4f8fae05158fe940bdf727db7de WatchSource:0}: Error finding container 877f384478fb31c291170b684bf6805a2500d4f8fae05158fe940bdf727db7de: Status 404 returned error can't find the container with id 877f384478fb31c291170b684bf6805a2500d4f8fae05158fe940bdf727db7de Nov 02 21:13:50.846781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:50.846713 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7c4469fd-r8dll" event={"ID":"6e0f3822-e583-4d83-bfe7-809d70dd0824","Type":"ContainerStarted","Data":"877f384478fb31c291170b684bf6805a2500d4f8fae05158fe940bdf727db7de"} Nov 02 21:13:51.740580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:51.740541 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:51.852275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:51.852232 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7c4469fd-r8dll" event={"ID":"6e0f3822-e583-4d83-bfe7-809d70dd0824","Type":"ContainerStarted","Data":"6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0"} Nov 02 21:13:51.867632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:51.867595 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:51.932664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:51.932635 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:51.932813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:51.932689 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:51.937765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:51.937741 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:52.011852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:52.011773 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c7c4469fd-r8dll" podStartSLOduration=2.011758924 podStartE2EDuration="2.011758924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:52.011251319 +0000 UTC m=+61.566879658" watchObservedRunningTime="2026-11-02 21:13:52.011758924 +0000 UTC m=+61.567387289" Nov 02 21:13:52.860795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:52.860751 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:13:53.159745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:53.159663 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:53.285318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:53.285287 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:55.153944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:55.153902 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:55.268055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:55.268017 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:56.369747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:56.369716 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:56.498640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:56.498587 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:59.771331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:13:59.771303 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:00.429489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:00.429454 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:14:00.429661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:00.429505 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:14:00.434283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:00.434264 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:14:00.889871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:00.889845 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:14:01.046170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:01.046138 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85887bddc7-vgf66"] Nov 02 21:14:06.736466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:06.736431 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:06.854638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:06.854608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:10.613459 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:14:10.613425 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:14:11.777298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:11.777269 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:20.704172 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:20.704141 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:20.826050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:20.826020 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:21.730269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:21.730237 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:21.848628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:21.848584 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:25.492959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:25.492921 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9cc7j_1f5e5ce7-b1e0-46a6-8170-c379721e6e45/serve-healthcheck-canary/0.log" Nov 02 21:14:26.065579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.065541 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85887bddc7-vgf66" podUID="f4a12c79-6df1-4f1f-bea0-1c32dce168f2" containerName="console" containerID="cri-o://cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d" gracePeriod=15 Nov 02 21:14:26.333249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.333188 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:14:26.453827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.453798 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-oauth-config\") pod \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " Nov 02 21:14:26.454008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.453832 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-config\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-config\") pod \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " Nov 02 21:14:26.454008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.453877 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-serving-cert\") pod \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " Nov 02 21:14:26.454008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.453905 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-tmp\") pod \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " Nov 02 21:14:26.454008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.453933 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-kube-api-access-sv7zs\" (UniqueName: \"kubernetes.io/projected/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-kube-api-access-sv7zs\") pod \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " Nov 02 21:14:26.454008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.453951 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-service-ca\") pod \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " Nov 02 21:14:26.454008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.453991 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-oauth-serving-cert\") pod \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\" (UID: \"f4a12c79-6df1-4f1f-bea0-1c32dce168f2\") " Nov 02 21:14:26.454323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.454257 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-tmp" pod "f4a12c79-6df1-4f1f-bea0-1c32dce168f2" (UID: "f4a12c79-6df1-4f1f-bea0-1c32dce168f2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:26.454443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.454422 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-config" pod "f4a12c79-6df1-4f1f-bea0-1c32dce168f2" (UID: "f4a12c79-6df1-4f1f-bea0-1c32dce168f2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:26.454504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.454458 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-oauth-serving-cert" pod "f4a12c79-6df1-4f1f-bea0-1c32dce168f2" (UID: "f4a12c79-6df1-4f1f-bea0-1c32dce168f2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:26.454575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.454547 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-service-ca" pod "f4a12c79-6df1-4f1f-bea0-1c32dce168f2" (UID: "f4a12c79-6df1-4f1f-bea0-1c32dce168f2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:26.456509 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.456476 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-oauth-config" pod "f4a12c79-6df1-4f1f-bea0-1c32dce168f2" (UID: "f4a12c79-6df1-4f1f-bea0-1c32dce168f2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:26.456631 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.456574 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-serving-cert" pod "f4a12c79-6df1-4f1f-bea0-1c32dce168f2" (UID: "f4a12c79-6df1-4f1f-bea0-1c32dce168f2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:26.456631 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.456587 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-kube-api-access-sv7zs" pod "f4a12c79-6df1-4f1f-bea0-1c32dce168f2" (UID: "f4a12c79-6df1-4f1f-bea0-1c32dce168f2"). InnerVolumeSpecName "kube-api-access-sv7zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:26.554933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.554910 4441 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-tmp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.554933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.554932 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv7zs\" (UniqueName: \"kubernetes.io/projected/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-kube-api-access-sv7zs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.555261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.554942 4441 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.555261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.554951 4441 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-oauth-serving-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.555261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.554960 4441 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-oauth-config\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.555261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.554978 4441 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-config\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.555261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.554987 4441 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4a12c79-6df1-4f1f-bea0-1c32dce168f2-console-serving-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:26.969528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.969442 4441 generic.go:356] "Generic (PLEG): container finished" podID="f4a12c79-6df1-4f1f-bea0-1c32dce168f2" containerID="cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d" exitCode=0 Nov 02 21:14:26.969528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.969510 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85887bddc7-vgf66" event={"ID":"f4a12c79-6df1-4f1f-bea0-1c32dce168f2","Type":"ContainerDied","Data":"cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d"} Nov 02 21:14:26.969528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.969520 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85887bddc7-vgf66" Nov 02 21:14:26.969811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.969553 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85887bddc7-vgf66" event={"ID":"f4a12c79-6df1-4f1f-bea0-1c32dce168f2","Type":"ContainerDied","Data":"11fa2d8108417ea6a0c866a6ccbb5cda9e7f45e5a0d114440cec8e949c65098f"} Nov 02 21:14:26.969811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.969584 4441 scope.go:122] "RemoveContainer" containerID="cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d" Nov 02 21:14:26.978781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.978757 4441 scope.go:122] "RemoveContainer" containerID="cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d" Nov 02 21:14:26.979145 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:14:26.979115 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d\": container with ID starting with cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d not found: ID does not exist" containerID="cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d" Nov 02 21:14:26.979263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:26.979150 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d"} err="failed to get container status \"cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d\": rpc error: code = NotFound desc = could not find container \"cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d\": container with ID starting with cc6a4ff9759a0e5cb05fd7cdd18e86b0fec0fa1723b04dac5a1e655e649ccf5d not found: ID does not exist" Nov 02 21:14:27.122572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:27.122548 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85887bddc7-vgf66"] Nov 02 21:14:27.169811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:27.169786 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85887bddc7-vgf66"] Nov 02 21:14:28.294602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:28.294563 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:28.580821 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:28.580791 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a12c79-6df1-4f1f-bea0-1c32dce168f2" path="/var/lib/kubelet/pods/f4a12c79-6df1-4f1f-bea0-1c32dce168f2/volumes" Nov 02 21:14:28.668726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:28.668691 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:31.573199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:31.573130 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:31.692661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:31.692632 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:40.613058 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:14:40.613020 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:14:41.094068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:41.094032 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:42.721058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.721021 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75f4fc578c-5gk77"] Nov 02 21:14:42.721441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.721256 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f4a12c79-6df1-4f1f-bea0-1c32dce168f2" containerName="console" Nov 02 21:14:42.721441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.721266 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a12c79-6df1-4f1f-bea0-1c32dce168f2" containerName="console" Nov 02 21:14:42.721441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.721319 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a12c79-6df1-4f1f-bea0-1c32dce168f2" containerName="console" Nov 02 21:14:42.723279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.723262 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.820650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.820612 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f4fc578c-5gk77"] Nov 02 21:14:42.866233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866204 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-config\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.866398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866237 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-service-ca\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.866398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866264 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-trusted-ca-bundle\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.866398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866319 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5f88b6-30ae-4990-9a7f-9e83b882df88-tmp\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.866398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866370 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5sx5\" (UniqueName: \"kubernetes.io/projected/bb5f88b6-30ae-4990-9a7f-9e83b882df88-kube-api-access-n5sx5\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.866565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866420 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-oauth-config\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.866565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866440 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-oauth-serving-cert\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.866565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.866456 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-serving-cert\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967389 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-config\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967427 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-service-ca\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967449 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-trusted-ca-bundle\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967557 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5f88b6-30ae-4990-9a7f-9e83b882df88-tmp\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967586 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5sx5\" (UniqueName: \"kubernetes.io/projected/bb5f88b6-30ae-4990-9a7f-9e83b882df88-kube-api-access-n5sx5\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967612 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-oauth-config\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967632 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-oauth-serving-cert\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.967728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.967648 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-serving-cert\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.968178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.968145 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5f88b6-30ae-4990-9a7f-9e83b882df88-tmp\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.968318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.968271 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-service-ca\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.968407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.968382 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-config\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.968472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.968407 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-oauth-serving-cert\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.968472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.968417 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-trusted-ca-bundle\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.970164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.970145 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-serving-cert\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.970332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.970310 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-oauth-config\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:42.993952 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:42.993896 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:43.020938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:43.020917 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5sx5\" (UniqueName: \"kubernetes.io/projected/bb5f88b6-30ae-4990-9a7f-9e83b882df88-kube-api-access-n5sx5\") pod \"console-75f4fc578c-5gk77\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:43.033724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:43.033700 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:43.108763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:43.108734 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:43.393246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:43.393208 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:43.446142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:43.446110 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f4fc578c-5gk77"] Nov 02 21:14:43.454610 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:14:43.454583 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5f88b6_30ae_4990_9a7f_9e83b882df88.slice/crio-3e334fe36bfa07bf311dbde8c016b50d4e4935de07ad990e266197470d15aa11 WatchSource:0}: Error finding container 3e334fe36bfa07bf311dbde8c016b50d4e4935de07ad990e266197470d15aa11: Status 404 returned error can't find the container with id 3e334fe36bfa07bf311dbde8c016b50d4e4935de07ad990e266197470d15aa11 Nov 02 21:14:44.030988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:44.030946 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f4fc578c-5gk77" event={"ID":"bb5f88b6-30ae-4990-9a7f-9e83b882df88","Type":"ContainerStarted","Data":"272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b"} Nov 02 21:14:44.031387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:44.031009 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f4fc578c-5gk77" event={"ID":"bb5f88b6-30ae-4990-9a7f-9e83b882df88","Type":"ContainerStarted","Data":"3e334fe36bfa07bf311dbde8c016b50d4e4935de07ad990e266197470d15aa11"} Nov 02 21:14:44.140891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:44.140842 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75f4fc578c-5gk77" podStartSLOduration=2.140828628 podStartE2EDuration="2.140828628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:44.140026148 +0000 UTC m=+113.695654488" watchObservedRunningTime="2026-11-02 21:14:44.140828628 +0000 UTC m=+113.696456964" Nov 02 21:14:53.034149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:53.034117 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:53.034575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:53.034175 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:53.038686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:53.038664 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:53.059854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:53.059834 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:14:53.282154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:14:53.282126 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c7c4469fd-r8dll"] Nov 02 21:15:06.102273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:06.102192 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:10.612507 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:15:10.612468 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:15:17.535325 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:17.535285 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:18.300786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.300730 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c7c4469fd-r8dll" podUID="6e0f3822-e583-4d83-bfe7-809d70dd0824" containerName="console" containerID="cri-o://6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0" gracePeriod=15 Nov 02 21:15:18.543785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.543761 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:15:18.624842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.624810 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-config\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-config\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.624848 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-serving-cert\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.624873 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-trusted-ca-bundle\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.624938 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-service-ca\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.624989 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0f3822-e583-4d83-bfe7-809d70dd0824-kube-api-access-4zqch\" (UniqueName: \"kubernetes.io/projected/6e0f3822-e583-4d83-bfe7-809d70dd0824-kube-api-access-4zqch\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625033 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-oauth-config\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625069 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e0f3822-e583-4d83-bfe7-809d70dd0824-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e0f3822-e583-4d83-bfe7-809d70dd0824-tmp\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625098 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-oauth-serving-cert\") pod \"6e0f3822-e583-4d83-bfe7-809d70dd0824\" (UID: \"6e0f3822-e583-4d83-bfe7-809d70dd0824\") " Nov 02 21:15:18.625295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625229 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-config" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:18.625497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625469 4441 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-config\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.625614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625489 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0f3822-e583-4d83-bfe7-809d70dd0824-tmp" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:18.625614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625560 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-oauth-serving-cert" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:18.625872 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625851 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-service-ca" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:18.625947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.625882 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-trusted-ca-bundle" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:18.627225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.627186 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-oauth-config" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:18.627311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.627248 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-serving-cert" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:18.627488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.627466 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0f3822-e583-4d83-bfe7-809d70dd0824-kube-api-access-4zqch" pod "6e0f3822-e583-4d83-bfe7-809d70dd0824" (UID: "6e0f3822-e583-4d83-bfe7-809d70dd0824"). InnerVolumeSpecName "kube-api-access-4zqch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:18.726532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.726503 4441 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-oauth-config\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.726532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.726528 4441 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e0f3822-e583-4d83-bfe7-809d70dd0824-tmp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.726532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.726538 4441 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-oauth-serving-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.726747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.726546 4441 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e0f3822-e583-4d83-bfe7-809d70dd0824-console-serving-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.726747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.726556 4441 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-trusted-ca-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.726747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.726567 4441 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0f3822-e583-4d83-bfe7-809d70dd0824-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.726747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:18.726575 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zqch\" (UniqueName: \"kubernetes.io/projected/6e0f3822-e583-4d83-bfe7-809d70dd0824-kube-api-access-4zqch\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:19.138100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.138068 4441 generic.go:356] "Generic (PLEG): container finished" podID="6e0f3822-e583-4d83-bfe7-809d70dd0824" containerID="6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0" exitCode=0 Nov 02 21:15:19.138254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.138112 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7c4469fd-r8dll" Nov 02 21:15:19.138254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.138122 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7c4469fd-r8dll" event={"ID":"6e0f3822-e583-4d83-bfe7-809d70dd0824","Type":"ContainerDied","Data":"6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0"} Nov 02 21:15:19.138254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.138167 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7c4469fd-r8dll" event={"ID":"6e0f3822-e583-4d83-bfe7-809d70dd0824","Type":"ContainerDied","Data":"877f384478fb31c291170b684bf6805a2500d4f8fae05158fe940bdf727db7de"} Nov 02 21:15:19.138254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.138187 4441 scope.go:122] "RemoveContainer" containerID="6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0" Nov 02 21:15:19.145530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.145422 4441 scope.go:122] "RemoveContainer" containerID="6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0" Nov 02 21:15:19.145742 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:15:19.145714 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0\": container with ID starting with 6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0 not found: ID does not exist" containerID="6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0" Nov 02 21:15:19.145863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.145748 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0"} err="failed to get container status \"6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0\": rpc error: code = NotFound desc = could not find container \"6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0\": container with ID starting with 6a2ec03135324b540814ad9f430e24ee27d1f24fa2cd8f4149c75d6b222997e0 not found: ID does not exist" Nov 02 21:15:19.292714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.292687 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c7c4469fd-r8dll"] Nov 02 21:15:19.339825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:19.339800 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c7c4469fd-r8dll"] Nov 02 21:15:20.580395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:20.580365 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0f3822-e583-4d83-bfe7-809d70dd0824" path="/var/lib/kubelet/pods/6e0f3822-e583-4d83-bfe7-809d70dd0824/volumes" Nov 02 21:15:40.613339 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:15:40.613299 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:15:41.430566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:41.430515 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:45.959065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:45.959028 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:46.074993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:46.074917 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:58.614771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:58.614727 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:58.741075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:58.741042 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:59.326037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.326009 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d59747dc-8jg8s"] Nov 02 21:15:59.326251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.326240 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6e0f3822-e583-4d83-bfe7-809d70dd0824" containerName="console" Nov 02 21:15:59.326301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.326252 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0f3822-e583-4d83-bfe7-809d70dd0824" containerName="console" Nov 02 21:15:59.326338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.326305 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0f3822-e583-4d83-bfe7-809d70dd0824" containerName="console" Nov 02 21:15:59.328716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.328701 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.398997 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-oauth-config\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.399030 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-oauth-serving-cert\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.399046 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-serving-cert\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.399060 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-config\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.399175 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-service-ca\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.399199 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-trusted-ca-bundle\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.399217 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw76v\" (UniqueName: \"kubernetes.io/projected/93bb6551-6bd4-4e0b-b493-13292b352ef4-kube-api-access-qw76v\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.399342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.399307 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bb6551-6bd4-4e0b-b493-13292b352ef4-tmp\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.425791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.425769 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d59747dc-8jg8s"] Nov 02 21:15:59.500426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500403 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-oauth-config\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.500572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500435 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-oauth-serving-cert\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.500572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500454 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-serving-cert\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.500572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500470 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-config\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.500572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500503 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-service-ca\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.500572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500546 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-trusted-ca-bundle\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.500825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500586 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw76v\" (UniqueName: \"kubernetes.io/projected/93bb6551-6bd4-4e0b-b493-13292b352ef4-kube-api-access-qw76v\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.500825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.500639 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bb6551-6bd4-4e0b-b493-13292b352ef4-tmp\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.501089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.501072 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bb6551-6bd4-4e0b-b493-13292b352ef4-tmp\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.501340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.501319 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-service-ca\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.501391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.501332 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-oauth-serving-cert\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.501391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.501319 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-config\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.502063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.502039 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93bb6551-6bd4-4e0b-b493-13292b352ef4-trusted-ca-bundle\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.502950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.502928 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-oauth-config\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.503148 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.503130 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93bb6551-6bd4-4e0b-b493-13292b352ef4-console-serving-cert\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.554169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.554149 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw76v\" (UniqueName: \"kubernetes.io/projected/93bb6551-6bd4-4e0b-b493-13292b352ef4-kube-api-access-qw76v\") pod \"console-69d59747dc-8jg8s\" (UID: \"93bb6551-6bd4-4e0b-b493-13292b352ef4\") " pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.597653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.597596 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:59.638660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.638628 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:15:59.717623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.717598 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:59.984234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:15:59.984171 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:00.054178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:00.054148 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d59747dc-8jg8s"] Nov 02 21:16:00.062369 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:16:00.062344 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bb6551_6bd4_4e0b_b493_13292b352ef4.slice/crio-f12563741cc37382b6ea67c976c61c2fbd72beb61f85a521605d39bd11dc0224 WatchSource:0}: Error finding container f12563741cc37382b6ea67c976c61c2fbd72beb61f85a521605d39bd11dc0224: Status 404 returned error can't find the container with id f12563741cc37382b6ea67c976c61c2fbd72beb61f85a521605d39bd11dc0224 Nov 02 21:16:00.101755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:00.101723 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:00.272574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:00.272492 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d59747dc-8jg8s" event={"ID":"93bb6551-6bd4-4e0b-b493-13292b352ef4","Type":"ContainerStarted","Data":"b3f8b4b1a92b93992460ef605f9947e9de206fb191fe7adb0e970fb86cc8b724"} Nov 02 21:16:00.272574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:00.272542 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d59747dc-8jg8s" event={"ID":"93bb6551-6bd4-4e0b-b493-13292b352ef4","Type":"ContainerStarted","Data":"f12563741cc37382b6ea67c976c61c2fbd72beb61f85a521605d39bd11dc0224"} Nov 02 21:16:00.380354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:00.380306 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69d59747dc-8jg8s" podStartSLOduration=1.380293974 podStartE2EDuration="1.380293974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:00.379507518 +0000 UTC m=+189.935135854" watchObservedRunningTime="2026-11-02 21:16:00.380293974 +0000 UTC m=+189.935922308" Nov 02 21:16:06.749881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:06.749848 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:06.872085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:06.872016 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:09.639666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:09.639623 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:16:09.639666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:09.639684 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:16:09.644436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:09.644416 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:16:10.310888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:10.310862 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69d59747dc-8jg8s" Nov 02 21:16:10.466434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:10.466406 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f4fc578c-5gk77"] Nov 02 21:16:10.613360 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:16:10.613289 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:16:23.223272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:23.223236 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:23.346519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:23.346486 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:35.485412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.485367 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-75f4fc578c-5gk77" podUID="bb5f88b6-30ae-4990-9a7f-9e83b882df88" containerName="console" containerID="cri-o://272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b" gracePeriod=15 Nov 02 21:16:35.719894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.719871 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:16:35.848132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848104 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-config\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-config\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848137 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-trusted-ca-bundle\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848169 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-oauth-config\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848196 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-oauth-serving-cert\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848246 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5f88b6-30ae-4990-9a7f-9e83b882df88-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5f88b6-30ae-4990-9a7f-9e83b882df88-tmp\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848308 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-service-ca\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848345 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5f88b6-30ae-4990-9a7f-9e83b882df88-kube-api-access-n5sx5\" (UniqueName: \"kubernetes.io/projected/bb5f88b6-30ae-4990-9a7f-9e83b882df88-kube-api-access-n5sx5\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848376 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-serving-cert\") pod \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\" (UID: \"bb5f88b6-30ae-4990-9a7f-9e83b882df88\") " Nov 02 21:16:35.848651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848580 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-config" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:16:35.848651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848585 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-trusted-ca-bundle" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:16:35.848651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848594 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5f88b6-30ae-4990-9a7f-9e83b882df88-tmp" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:16:35.848742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848671 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-service-ca" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:16:35.848989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.848956 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-oauth-serving-cert" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:16:35.850423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.850391 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5f88b6-30ae-4990-9a7f-9e83b882df88-kube-api-access-n5sx5" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "kube-api-access-n5sx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:16:35.850555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.850532 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-serving-cert" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:16:35.850631 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.850616 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-oauth-config" pod "bb5f88b6-30ae-4990-9a7f-9e83b882df88" (UID: "bb5f88b6-30ae-4990-9a7f-9e83b882df88"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:16:35.949721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949699 4441 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-config\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:35.949721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949720 4441 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-trusted-ca-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:35.949841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949730 4441 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-oauth-config\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:35.949841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949739 4441 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-oauth-serving-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:35.949841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949748 4441 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5f88b6-30ae-4990-9a7f-9e83b882df88-tmp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:35.949841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949757 4441 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f88b6-30ae-4990-9a7f-9e83b882df88-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:35.949841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949766 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5sx5\" (UniqueName: \"kubernetes.io/projected/bb5f88b6-30ae-4990-9a7f-9e83b882df88-kube-api-access-n5sx5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:35.949841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:35.949775 4441 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5f88b6-30ae-4990-9a7f-9e83b882df88-console-serving-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:36.386358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.386328 4441 generic.go:356] "Generic (PLEG): container finished" podID="bb5f88b6-30ae-4990-9a7f-9e83b882df88" containerID="272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b" exitCode=0 Nov 02 21:16:36.386493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.386364 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f4fc578c-5gk77" event={"ID":"bb5f88b6-30ae-4990-9a7f-9e83b882df88","Type":"ContainerDied","Data":"272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b"} Nov 02 21:16:36.386493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.386374 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f4fc578c-5gk77" Nov 02 21:16:36.386493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.386397 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f4fc578c-5gk77" event={"ID":"bb5f88b6-30ae-4990-9a7f-9e83b882df88","Type":"ContainerDied","Data":"3e334fe36bfa07bf311dbde8c016b50d4e4935de07ad990e266197470d15aa11"} Nov 02 21:16:36.386493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.386416 4441 scope.go:122] "RemoveContainer" containerID="272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b" Nov 02 21:16:36.394329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.394310 4441 scope.go:122] "RemoveContainer" containerID="272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b" Nov 02 21:16:36.394583 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:16:36.394564 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b\": container with ID starting with 272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b not found: ID does not exist" containerID="272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b" Nov 02 21:16:36.394658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.394597 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b"} err="failed to get container status \"272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b\": rpc error: code = NotFound desc = could not find container \"272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b\": container with ID starting with 272e094411b0e4315f6dd382ca9eefdb37b26ae69bbaf61427ec43ad7a25d60b not found: ID does not exist" Nov 02 21:16:36.541298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.541272 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f4fc578c-5gk77"] Nov 02 21:16:36.570605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.570585 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75f4fc578c-5gk77"] Nov 02 21:16:36.580702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:36.580671 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5f88b6-30ae-4990-9a7f-9e83b882df88" path="/var/lib/kubelet/pods/bb5f88b6-30ae-4990-9a7f-9e83b882df88/volumes" Nov 02 21:16:40.612651 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:16:40.612602 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:16:41.768413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:16:41.768374 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:10.612810 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:17:10.612775 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:17:26.732787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:26.732748 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:28.123029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.122992 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:28.260224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.260193 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:28.625064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.625033 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh"] Nov 02 21:17:28.625289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.625277 4441 cpu_manager.go:414] "containerMap: removing container" podUID="bb5f88b6-30ae-4990-9a7f-9e83b882df88" containerName="console" Nov 02 21:17:28.625331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.625289 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5f88b6-30ae-4990-9a7f-9e83b882df88" containerName="console" Nov 02 21:17:28.625369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.625355 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5f88b6-30ae-4990-9a7f-9e83b882df88" containerName="console" Nov 02 21:17:28.627334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.627318 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.692397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.692375 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.692546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.692411 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.692546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.692511 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkmg\" (UniqueName: \"kubernetes.io/projected/14741067-19bd-4053-b3c3-7a3211d8222f-kube-api-access-xkkmg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.724155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.724130 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 21:17:28.724267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.724163 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 21:17:28.724267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.724163 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6bc4z\"" Nov 02 21:17:28.725285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.725260 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh"] Nov 02 21:17:28.726281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.726256 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6"] Nov 02 21:17:28.729660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.729640 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.778880 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.778853 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6"] Nov 02 21:17:28.779904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.779882 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-c7g2c\"" Nov 02 21:17:28.793419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793394 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkmg\" (UniqueName: \"kubernetes.io/projected/14741067-19bd-4053-b3c3-7a3211d8222f-kube-api-access-xkkmg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.793519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793434 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.793519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793452 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.793519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793470 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.793682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793581 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw8vz\" (UniqueName: \"kubernetes.io/projected/7720b08e-742a-4af9-9dd8-deb90a7d084e-kube-api-access-kw8vz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.793682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793634 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.793817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793778 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.793932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.793913 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.822702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.822679 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs"] Nov 02 21:17:28.824915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.824900 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.846831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.846809 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkmg\" (UniqueName: \"kubernetes.io/projected/14741067-19bd-4053-b3c3-7a3211d8222f-kube-api-access-xkkmg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.879643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.879597 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs"] Nov 02 21:17:28.880033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.880007 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-s8jkn\"" Nov 02 21:17:28.894685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.894666 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnf9n\" (UniqueName: \"kubernetes.io/projected/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-kube-api-access-nnf9n\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.894775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.894694 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.894775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.894711 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.894775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.894740 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.894872 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.894798 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.894907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.894884 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8vz\" (UniqueName: \"kubernetes.io/projected/7720b08e-742a-4af9-9dd8-deb90a7d084e-kube-api-access-kw8vz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.895041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.895026 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.895081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.895064 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.922339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.922319 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5"] Nov 02 21:17:28.924586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.924569 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:28.939267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.939247 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:17:28.948340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.948320 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw8vz\" (UniqueName: \"kubernetes.io/projected/7720b08e-742a-4af9-9dd8-deb90a7d084e-kube-api-access-kw8vz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:28.980641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.980599 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5"] Nov 02 21:17:28.980901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.980884 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5dqjd\"" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.995445 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnf9n\" (UniqueName: \"kubernetes.io/projected/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-kube-api-access-nnf9n\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.995503 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.995534 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.995559 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.995595 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/07f487c3-36be-4358-9696-d1b1ce4e453f-kube-api-access-zmm4c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.995623 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.996338 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:28.996436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:28.996393 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:29.042162 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.042137 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:17:29.049886 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.049861 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnf9n\" (UniqueName: \"kubernetes.io/projected/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-kube-api-access-nnf9n\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:29.055765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.055737 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:29.096340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.096312 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:29.096455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.096367 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/07f487c3-36be-4358-9696-d1b1ce4e453f-kube-api-access-zmm4c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:29.096455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.096398 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:29.096735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.096713 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:29.096815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.096802 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:29.136518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.136459 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:17:29.151466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.151446 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/07f487c3-36be-4358-9696-d1b1ce4e453f-kube-api-access-zmm4c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:29.237173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.237141 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:17:29.241991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.241953 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn"] Nov 02 21:17:29.244817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.244796 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.298331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.298302 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-fx7m6\"" Nov 02 21:17:29.298453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.298370 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.298520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.298457 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.298520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.298504 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8md\" (UniqueName: \"kubernetes.io/projected/1f17b360-91e1-492f-bdf0-de861ce4dd43-kube-api-access-6g8md\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.300265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.300240 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn"] Nov 02 21:17:29.388990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.388916 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh"] Nov 02 21:17:29.399330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.399309 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.399434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.399338 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.399434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.399358 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8md\" (UniqueName: \"kubernetes.io/projected/1f17b360-91e1-492f-bdf0-de861ce4dd43-kube-api-access-6g8md\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.399761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.399739 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.399792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.399747 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.415501 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:29.415478 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14741067_19bd_4053_b3c3_7a3211d8222f.slice/crio-94abb758b489ec08c5b208d754e1ad07160e4baa8f20735a07b795b663dd6b25 WatchSource:0}: Error finding container 94abb758b489ec08c5b208d754e1ad07160e4baa8f20735a07b795b663dd6b25: Status 404 returned error can't find the container with id 94abb758b489ec08c5b208d754e1ad07160e4baa8f20735a07b795b663dd6b25 Nov 02 21:17:29.420116 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.420096 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6"] Nov 02 21:17:29.427118 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:29.427093 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7720b08e_742a_4af9_9dd8_deb90a7d084e.slice/crio-8fac0255710ac0047b2e5adf87e8651a9833f3b831845d49c1ab44f9a3e6f9c5 WatchSource:0}: Error finding container 8fac0255710ac0047b2e5adf87e8651a9833f3b831845d49c1ab44f9a3e6f9c5: Status 404 returned error can't find the container with id 8fac0255710ac0047b2e5adf87e8651a9833f3b831845d49c1ab44f9a3e6f9c5 Nov 02 21:17:29.452323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.452299 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8md\" (UniqueName: \"kubernetes.io/projected/1f17b360-91e1-492f-bdf0-de861ce4dd43-kube-api-access-6g8md\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.507365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.507344 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs"] Nov 02 21:17:29.514516 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:29.514488 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231e8844_6b1f_4f79_a951_bf0e2b8a93d9.slice/crio-6276357fc99158b0c1708ca5f55f97e4a651637c186fde25f924e88251ead9e2 WatchSource:0}: Error finding container 6276357fc99158b0c1708ca5f55f97e4a651637c186fde25f924e88251ead9e2: Status 404 returned error can't find the container with id 6276357fc99158b0c1708ca5f55f97e4a651637c186fde25f924e88251ead9e2 Nov 02 21:17:29.557360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.557333 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" event={"ID":"14741067-19bd-4053-b3c3-7a3211d8222f","Type":"ContainerStarted","Data":"94abb758b489ec08c5b208d754e1ad07160e4baa8f20735a07b795b663dd6b25"} Nov 02 21:17:29.558382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.558358 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" event={"ID":"7720b08e-742a-4af9-9dd8-deb90a7d084e","Type":"ContainerStarted","Data":"8fac0255710ac0047b2e5adf87e8651a9833f3b831845d49c1ab44f9a3e6f9c5"} Nov 02 21:17:29.558641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.558627 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:17:29.559398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.559380 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" event={"ID":"231e8844-6b1f-4f79-a951-bf0e2b8a93d9","Type":"ContainerStarted","Data":"6276357fc99158b0c1708ca5f55f97e4a651637c186fde25f924e88251ead9e2"} Nov 02 21:17:29.605334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.605309 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5"] Nov 02 21:17:29.919696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:29.919660 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn"] Nov 02 21:17:29.927484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:29.927453 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f17b360_91e1_492f_bdf0_de861ce4dd43.slice/crio-724481a8b7e821feb1469a7978d78b5f1dc95157363b245180bbbb4a62122439 WatchSource:0}: Error finding container 724481a8b7e821feb1469a7978d78b5f1dc95157363b245180bbbb4a62122439: Status 404 returned error can't find the container with id 724481a8b7e821feb1469a7978d78b5f1dc95157363b245180bbbb4a62122439 Nov 02 21:17:30.565507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:30.565425 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" event={"ID":"1f17b360-91e1-492f-bdf0-de861ce4dd43","Type":"ContainerStarted","Data":"724481a8b7e821feb1469a7978d78b5f1dc95157363b245180bbbb4a62122439"} Nov 02 21:17:30.567338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:30.567280 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" event={"ID":"07f487c3-36be-4358-9696-d1b1ce4e453f","Type":"ContainerStarted","Data":"70f1f1091fd6d5cd34d58bef1cd8823c22a08f2791d945fb5b0b0eb1f3f3b6df"} Nov 02 21:17:31.146187 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.146154 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k"] Nov 02 21:17:31.149107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.149082 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.215356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.215319 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.215517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.215376 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9jw\" (UniqueName: \"kubernetes.io/projected/f5179f5c-ef2c-4687-8be2-922114d4433a-kube-api-access-jn9jw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.215517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.215414 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.248961 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.248924 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-vxw9r\"" Nov 02 21:17:31.250259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.250163 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k"] Nov 02 21:17:31.315928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.315879 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.316127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.315939 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9jw\" (UniqueName: \"kubernetes.io/projected/f5179f5c-ef2c-4687-8be2-922114d4433a-kube-api-access-jn9jw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.316127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.315992 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.316440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.316419 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.316693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.316674 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.374082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.374018 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9jw\" (UniqueName: \"kubernetes.io/projected/f5179f5c-ef2c-4687-8be2-922114d4433a-kube-api-access-jn9jw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.469198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.468881 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:17:31.863994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:31.859437 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k"] Nov 02 21:17:31.870232 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:31.870193 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5179f5c_ef2c_4687_8be2_922114d4433a.slice/crio-bd4c8abcf7854504d040a9c013fd371d01853aa8d97e2308def1fdce35b04c0e WatchSource:0}: Error finding container bd4c8abcf7854504d040a9c013fd371d01853aa8d97e2308def1fdce35b04c0e: Status 404 returned error can't find the container with id bd4c8abcf7854504d040a9c013fd371d01853aa8d97e2308def1fdce35b04c0e Nov 02 21:17:32.587239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:32.587197 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" event={"ID":"f5179f5c-ef2c-4687-8be2-922114d4433a","Type":"ContainerStarted","Data":"bd4c8abcf7854504d040a9c013fd371d01853aa8d97e2308def1fdce35b04c0e"} Nov 02 21:17:33.318007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.317779 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h"] Nov 02 21:17:33.321601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.321580 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.374534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.374492 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-dd8t7\"" Nov 02 21:17:33.382121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.382095 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h"] Nov 02 21:17:33.442558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.442525 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.442726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.442596 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.442726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.442640 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgw6\" (UniqueName: \"kubernetes.io/projected/9717bd05-9191-42c8-b22d-d7771695c05f-kube-api-access-ptgw6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.545313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.544085 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgw6\" (UniqueName: \"kubernetes.io/projected/9717bd05-9191-42c8-b22d-d7771695c05f-kube-api-access-ptgw6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.545313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.544200 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.545313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.544227 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.545313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.544670 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.545313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.545262 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.620565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.620482 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgw6\" (UniqueName: \"kubernetes.io/projected/9717bd05-9191-42c8-b22d-d7771695c05f-kube-api-access-ptgw6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:33.639314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:33.639287 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:17:34.075246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:34.075210 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h"] Nov 02 21:17:34.609517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:34.609465 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" event={"ID":"9717bd05-9191-42c8-b22d-d7771695c05f","Type":"ContainerStarted","Data":"cf871b1adc688d2c3e6fa66f35fa1acc275df48410326146310eeb7c1f43da2c"} Nov 02 21:17:34.824754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:34.824670 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:34.948398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:34.948242 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:35.332303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.330659 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw"] Nov 02 21:17:35.338218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.333888 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362606 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362652 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362698 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362761 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362788 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362825 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58bz\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-kube-api-access-m58bz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362854 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362881 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.363107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.362906 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.384584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.384450 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 21:17:35.384584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.384550 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:17:35.405817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.405759 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:17:35.410977 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.410895 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:35.434075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.434041 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw"] Nov 02 21:17:35.434295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.434190 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-445mt\"" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464076 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464120 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464173 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58bz\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-kube-api-access-m58bz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464209 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464237 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464260 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464299 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464333 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464378 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.465213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.464802 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.466979 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.466065 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.466979 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.466603 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.467138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.466956 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.467138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.467043 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.469835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.469814 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.476733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.476688 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.529376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.529342 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.533437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.533411 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58bz\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-kube-api-access-m58bz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wkbnw\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.623611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.623522 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:35.629646 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.629618 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth"] Nov 02 21:17:35.633267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.633242 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.654904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.654878 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:35.665843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.665814 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.665945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.665858 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.665945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.665893 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.665945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.665920 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzp2p\" (UniqueName: \"kubernetes.io/projected/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-kube-api-access-lzp2p\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.666196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.665956 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.666196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.666052 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.666196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.666090 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.666196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.666126 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.666196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.666157 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.689256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.685462 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-2lmkj\"" Nov 02 21:17:35.693763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.693637 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth"] Nov 02 21:17:35.754064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.752952 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr"] Nov 02 21:17:35.757934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.757903 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.772126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.772098 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:35.775400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775365 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775428 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775467 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775496 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775532 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775555 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775584 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775608 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzp2p\" (UniqueName: \"kubernetes.io/projected/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-kube-api-access-lzp2p\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.775793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.775697 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.776200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.776154 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.783467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.783441 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q"] Nov 02 21:17:35.786570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.786549 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.786736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.786713 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.790043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.789958 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.790565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.790316 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.790565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.790416 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.790768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.790564 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.806819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.806801 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.811988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.811229 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625"] Nov 02 21:17:35.815048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.815017 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr"] Nov 02 21:17:35.815215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.815147 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.837327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.837304 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q"] Nov 02 21:17:35.843326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.843272 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.844226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.844203 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz"] Nov 02 21:17:35.847236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.847216 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:35.847412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.847216 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzp2p\" (UniqueName: \"kubernetes.io/projected/edd2ef9b-7d22-468c-8eeb-bf7161bbab07-kube-api-access-lzp2p\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth\" (UID: \"edd2ef9b-7d22-468c-8eeb-bf7161bbab07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.873014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.871850 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625"] Nov 02 21:17:35.876529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876464 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.876529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876515 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.876702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876602 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slcq\" (UniqueName: \"kubernetes.io/projected/a1957155-7229-4c75-a611-b020695ad17f-kube-api-access-7slcq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.876702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876664 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.876817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876736 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.876817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876785 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.876930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876823 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhr26\" (UniqueName: \"kubernetes.io/projected/f304c488-3134-4049-9f5b-39da4ef4a217-kube-api-access-dhr26\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.876930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876857 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.876930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.876888 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2fx\" (UniqueName: \"kubernetes.io/projected/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-kube-api-access-cm2fx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.928381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.928352 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz"] Nov 02 21:17:35.950817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.950790 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:35.977320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977288 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.977441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977339 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2fx\" (UniqueName: \"kubernetes.io/projected/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-kube-api-access-cm2fx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.977441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977387 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhwk\" (UniqueName: \"kubernetes.io/projected/b0d1e821-a731-477d-90ff-53a3fd32c8ab-kube-api-access-bzhwk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:35.977441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977437 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.977609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977457 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:35.977609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977474 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.977609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977507 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slcq\" (UniqueName: \"kubernetes.io/projected/a1957155-7229-4c75-a611-b020695ad17f-kube-api-access-7slcq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.977609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977537 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.977609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977559 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.977845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977623 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.977845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977651 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhr26\" (UniqueName: \"kubernetes.io/projected/f304c488-3134-4049-9f5b-39da4ef4a217-kube-api-access-dhr26\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.977845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.977668 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:35.978209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.978185 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:35.978895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.978870 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.979280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.979259 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:35.979593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.979568 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.979666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.979588 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:35.980052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:35.980032 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:36.034982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.034940 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhr26\" (UniqueName: \"kubernetes.io/projected/f304c488-3134-4049-9f5b-39da4ef4a217-kube-api-access-dhr26\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:36.051901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.051828 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2fx\" (UniqueName: \"kubernetes.io/projected/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-kube-api-access-cm2fx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:36.054773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.053548 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slcq\" (UniqueName: \"kubernetes.io/projected/a1957155-7229-4c75-a611-b020695ad17f-kube-api-access-7slcq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:36.078545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.078516 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:36.078663 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.078613 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:36.078721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.078692 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhwk\" (UniqueName: \"kubernetes.io/projected/b0d1e821-a731-477d-90ff-53a3fd32c8ab-kube-api-access-bzhwk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:36.079410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.079386 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:36.079647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.079627 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:36.094982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.094941 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw"] Nov 02 21:17:36.099534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.099300 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:17:36.106774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.106390 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:17:36.127915 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:36.127848 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf8cbb8_f7f8_4fd4_882c_3b9cc4307190.slice/crio-c2c5ab65cc32db13a4e863a6988849b65bc45c330e6d6d6a588663c217c79135 WatchSource:0}: Error finding container c2c5ab65cc32db13a4e863a6988849b65bc45c330e6d6d6a588663c217c79135: Status 404 returned error can't find the container with id c2c5ab65cc32db13a4e863a6988849b65bc45c330e6d6d6a588663c217c79135 Nov 02 21:17:36.135786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.135587 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:17:36.140206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.140156 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhwk\" (UniqueName: \"kubernetes.io/projected/b0d1e821-a731-477d-90ff-53a3fd32c8ab-kube-api-access-bzhwk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:36.166260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.166046 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:17:36.408634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.408538 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth"] Nov 02 21:17:36.543446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.543415 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q"] Nov 02 21:17:36.575331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.575306 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr"] Nov 02 21:17:36.595928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.595899 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz"] Nov 02 21:17:36.605651 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:36.605627 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d1e821_a731_477d_90ff_53a3fd32c8ab.slice/crio-4a7431ea9ff7002b4e78b7300fc53e2708aa5de66ab93407d4b792987c45586d WatchSource:0}: Error finding container 4a7431ea9ff7002b4e78b7300fc53e2708aa5de66ab93407d4b792987c45586d: Status 404 returned error can't find the container with id 4a7431ea9ff7002b4e78b7300fc53e2708aa5de66ab93407d4b792987c45586d Nov 02 21:17:36.623848 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.622032 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625"] Nov 02 21:17:36.639384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.630547 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" event={"ID":"f304c488-3134-4049-9f5b-39da4ef4a217","Type":"ContainerStarted","Data":"310670cef1bb0d656378367e8d345373f3a0a25f07a47a32fbbc84e85f7c5129"} Nov 02 21:17:36.639384 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:17:36.631094 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1957155_7229_4c75_a611_b020695ad17f.slice/crio-0780c37ea5d1afd460338edb657c34229a6439a00354512acfaa5c7cfcf0629b WatchSource:0}: Error finding container 0780c37ea5d1afd460338edb657c34229a6439a00354512acfaa5c7cfcf0629b: Status 404 returned error can't find the container with id 0780c37ea5d1afd460338edb657c34229a6439a00354512acfaa5c7cfcf0629b Nov 02 21:17:36.639384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.633551 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" event={"ID":"441de62c-792a-4bb0-ad70-d6ecc84bd5ad","Type":"ContainerStarted","Data":"35202b31c48589bc1606656f12c0be8c308c92c705053ddfed13e087be4b0da7"} Nov 02 21:17:36.639384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.638108 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" event={"ID":"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190","Type":"ContainerStarted","Data":"c2c5ab65cc32db13a4e863a6988849b65bc45c330e6d6d6a588663c217c79135"} Nov 02 21:17:36.641079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.640962 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" event={"ID":"edd2ef9b-7d22-468c-8eeb-bf7161bbab07","Type":"ContainerStarted","Data":"65a980462df73b3128da62e3e18ed776b20601cfdc380c1c6eff33d76c4976bf"} Nov 02 21:17:36.648159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:36.648108 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" event={"ID":"b0d1e821-a731-477d-90ff-53a3fd32c8ab","Type":"ContainerStarted","Data":"4a7431ea9ff7002b4e78b7300fc53e2708aa5de66ab93407d4b792987c45586d"} Nov 02 21:17:37.663799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:37.663757 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" event={"ID":"a1957155-7229-4c75-a611-b020695ad17f","Type":"ContainerStarted","Data":"0780c37ea5d1afd460338edb657c34229a6439a00354512acfaa5c7cfcf0629b"} Nov 02 21:17:39.723317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:39.723199 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:40.661929 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:17:40.661844 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:17:42.105232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:42.105171 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:49.546264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.546228 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:17:49.546678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.546322 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:17:49.546678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.546363 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:17:49.557706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.557145 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:17:49.557706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.557221 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:17:49.557706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.557265 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:17:49.811256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.811225 4441 generic.go:356] "Generic (PLEG): container finished" podID="14741067-19bd-4053-b3c3-7a3211d8222f" containerID="df04d4b0366c80815cbc95789c689de766e385e1e1218c629f83b4806de6e485" exitCode=0 Nov 02 21:17:49.811387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.811363 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" event={"ID":"14741067-19bd-4053-b3c3-7a3211d8222f","Type":"ContainerDied","Data":"df04d4b0366c80815cbc95789c689de766e385e1e1218c629f83b4806de6e485"} Nov 02 21:17:49.820103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:49.820050 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" event={"ID":"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190","Type":"ContainerStarted","Data":"47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b"} Nov 02 21:17:50.069892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.069837 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" podStartSLOduration=1.6592262930000001 podStartE2EDuration="15.069817004s" totalImagesPullingTime="13.410590711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:17:50.067947212 +0000 UTC m=+299.623575565" watchObservedRunningTime="2026-11-02 21:17:50.069817004 +0000 UTC m=+299.625445344" Nov 02 21:17:50.595623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.595592 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:17:50.597208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.597186 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:17:50.655764 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.655707 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:50.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.655939 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:50.661826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.661797 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:50.826927 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.826901 4441 generic.go:356] "Generic (PLEG): container finished" podID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerID="8d0d7069fd4d3d845da03a80ade826f90e729f90c646df2d6283d5cd634f6963" exitCode=0 Nov 02 21:17:50.827063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.827026 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" event={"ID":"7720b08e-742a-4af9-9dd8-deb90a7d084e","Type":"ContainerDied","Data":"8d0d7069fd4d3d845da03a80ade826f90e729f90c646df2d6283d5cd634f6963"} Nov 02 21:17:50.828948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.828921 4441 generic.go:356] "Generic (PLEG): container finished" podID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerID="475d09d64624d18c53fb4a8985e925073b94d15f3f40786aa52449fcda4726b5" exitCode=0 Nov 02 21:17:50.829059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.829024 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" event={"ID":"231e8844-6b1f-4f79-a951-bf0e2b8a93d9","Type":"ContainerDied","Data":"475d09d64624d18c53fb4a8985e925073b94d15f3f40786aa52449fcda4726b5"} Nov 02 21:17:50.830930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.830910 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" event={"ID":"edd2ef9b-7d22-468c-8eeb-bf7161bbab07","Type":"ContainerStarted","Data":"67c613968a7f19f908090443c4a6b0ddf6a01933eb0c964b8f118ae55f775008"} Nov 02 21:17:50.832818 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.832792 4441 generic.go:356] "Generic (PLEG): container finished" podID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerID="62fc041cb0aac1a4112de56876b1484bc2e1a634c12b02c8d4745bcda3ad23f2" exitCode=0 Nov 02 21:17:50.833051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.833029 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" event={"ID":"07f487c3-36be-4358-9696-d1b1ce4e453f","Type":"ContainerDied","Data":"62fc041cb0aac1a4112de56876b1484bc2e1a634c12b02c8d4745bcda3ad23f2"} Nov 02 21:17:50.834425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.834400 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:17:50.951628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.951600 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:50.951754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.951667 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:50.956851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:50.956828 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:51.261935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.261884 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" podStartSLOduration=3.132626402 podStartE2EDuration="16.261867984s" totalImagesPullingTime="13.129241582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:17:51.259582551 +0000 UTC m=+300.815210898" watchObservedRunningTime="2026-11-02 21:17:51.261867984 +0000 UTC m=+300.817496377" Nov 02 21:17:51.839632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.839599 4441 generic.go:356] "Generic (PLEG): container finished" podID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerID="f0d5af276fcee8f74d2dcc3a94380d355be5c799b2238389ab0ab6d8a6be427e" exitCode=0 Nov 02 21:17:51.840062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.839683 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" event={"ID":"1f17b360-91e1-492f-bdf0-de861ce4dd43","Type":"ContainerDied","Data":"f0d5af276fcee8f74d2dcc3a94380d355be5c799b2238389ab0ab6d8a6be427e"} Nov 02 21:17:51.842443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.842416 4441 generic.go:356] "Generic (PLEG): container finished" podID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerID="cc6b3455658fbc93d27ec76cbd93b190f0d7404d6bae75935272787046fbc54a" exitCode=0 Nov 02 21:17:51.842564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.842517 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" event={"ID":"b0d1e821-a731-477d-90ff-53a3fd32c8ab","Type":"ContainerDied","Data":"cc6b3455658fbc93d27ec76cbd93b190f0d7404d6bae75935272787046fbc54a"} Nov 02 21:17:51.844416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.844399 4441 generic.go:356] "Generic (PLEG): container finished" podID="f304c488-3134-4049-9f5b-39da4ef4a217" containerID="55e916374f808521e4515a233884ed278aced060d2e41f911c800ed593e8b666" exitCode=0 Nov 02 21:17:51.844503 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.844474 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" event={"ID":"f304c488-3134-4049-9f5b-39da4ef4a217","Type":"ContainerDied","Data":"55e916374f808521e4515a233884ed278aced060d2e41f911c800ed593e8b666"} Nov 02 21:17:51.846331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.846298 4441 generic.go:356] "Generic (PLEG): container finished" podID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerID="5688876eedfe68a3d47c9adba204d865853afcf623907d18b7b7ab0d96961087" exitCode=0 Nov 02 21:17:51.846729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.846684 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" event={"ID":"f5179f5c-ef2c-4687-8be2-922114d4433a","Type":"ContainerDied","Data":"5688876eedfe68a3d47c9adba204d865853afcf623907d18b7b7ab0d96961087"} Nov 02 21:17:51.848515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.848376 4441 generic.go:356] "Generic (PLEG): container finished" podID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerID="872a62ad264f3d950754e2df559e2dc35ef06be7c90720bb762f42e51abfca87" exitCode=0 Nov 02 21:17:51.848645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.848610 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" event={"ID":"441de62c-792a-4bb0-ad70-d6ecc84bd5ad","Type":"ContainerDied","Data":"872a62ad264f3d950754e2df559e2dc35ef06be7c90720bb762f42e51abfca87"} Nov 02 21:17:51.850555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.850534 4441 generic.go:356] "Generic (PLEG): container finished" podID="a1957155-7229-4c75-a611-b020695ad17f" containerID="871e9fa02df734c25fcf542d2ea37baf63f9eb45791bf2706bd68959ca9b9db0" exitCode=0 Nov 02 21:17:51.850807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.850787 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" event={"ID":"a1957155-7229-4c75-a611-b020695ad17f","Type":"ContainerDied","Data":"871e9fa02df734c25fcf542d2ea37baf63f9eb45791bf2706bd68959ca9b9db0"} Nov 02 21:17:51.852677 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:51.852655 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth" Nov 02 21:17:52.857092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:52.856952 4441 generic.go:356] "Generic (PLEG): container finished" podID="9717bd05-9191-42c8-b22d-d7771695c05f" containerID="7e374b16512c283c14afdecfa9196b33ce0ba0469234ad24901f1b7ee8583d1c" exitCode=0 Nov 02 21:17:52.857435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:52.857378 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" event={"ID":"9717bd05-9191-42c8-b22d-d7771695c05f","Type":"ContainerDied","Data":"7e374b16512c283c14afdecfa9196b33ce0ba0469234ad24901f1b7ee8583d1c"} Nov 02 21:17:52.860581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:52.860560 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:17:53.868784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.868693 4441 generic.go:356] "Generic (PLEG): container finished" podID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerID="bf585e6af16c38e2d485ccc9e852121d225d927f0e4321df39b47c86c351b0c0" exitCode=0 Nov 02 21:17:53.869255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.868820 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" event={"ID":"07f487c3-36be-4358-9696-d1b1ce4e453f","Type":"ContainerDied","Data":"bf585e6af16c38e2d485ccc9e852121d225d927f0e4321df39b47c86c351b0c0"} Nov 02 21:17:53.871239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.871209 4441 generic.go:356] "Generic (PLEG): container finished" podID="f304c488-3134-4049-9f5b-39da4ef4a217" containerID="dea94c8ccd01c6cbabd3b994b49bcd6ebc59b96e47f972208bf1bac3bab87b4d" exitCode=0 Nov 02 21:17:53.871328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.871282 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" event={"ID":"f304c488-3134-4049-9f5b-39da4ef4a217","Type":"ContainerDied","Data":"dea94c8ccd01c6cbabd3b994b49bcd6ebc59b96e47f972208bf1bac3bab87b4d"} Nov 02 21:17:53.874065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.874039 4441 generic.go:356] "Generic (PLEG): container finished" podID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerID="debd3fba0e503af6f2a5236b53d47b67aff8199299901fdba806daba313aa94e" exitCode=0 Nov 02 21:17:53.874150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.874123 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" event={"ID":"441de62c-792a-4bb0-ad70-d6ecc84bd5ad","Type":"ContainerDied","Data":"debd3fba0e503af6f2a5236b53d47b67aff8199299901fdba806daba313aa94e"} Nov 02 21:17:53.876701 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.876681 4441 generic.go:356] "Generic (PLEG): container finished" podID="a1957155-7229-4c75-a611-b020695ad17f" containerID="bb8a56b98b885d44682369197707099e925de1aae9fcf7efa1df60dc8bd45749" exitCode=0 Nov 02 21:17:53.876783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.876760 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" event={"ID":"a1957155-7229-4c75-a611-b020695ad17f","Type":"ContainerDied","Data":"bb8a56b98b885d44682369197707099e925de1aae9fcf7efa1df60dc8bd45749"} Nov 02 21:17:53.878838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.878813 4441 generic.go:356] "Generic (PLEG): container finished" podID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerID="ae90ad0903b8fc66f2d6cff7e03517c8382c4e36b26fd75bfe49e17a9f1c500c" exitCode=0 Nov 02 21:17:53.878942 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.878905 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" event={"ID":"7720b08e-742a-4af9-9dd8-deb90a7d084e","Type":"ContainerDied","Data":"ae90ad0903b8fc66f2d6cff7e03517c8382c4e36b26fd75bfe49e17a9f1c500c"} Nov 02 21:17:53.880934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.880909 4441 generic.go:356] "Generic (PLEG): container finished" podID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerID="796c6d0e7481b8c8100c56d0c3e1236fb832987120c0ea1dc9e46c98c1d56bb2" exitCode=0 Nov 02 21:17:53.881047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.880988 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" event={"ID":"231e8844-6b1f-4f79-a951-bf0e2b8a93d9","Type":"ContainerDied","Data":"796c6d0e7481b8c8100c56d0c3e1236fb832987120c0ea1dc9e46c98c1d56bb2"} Nov 02 21:17:53.883070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.883049 4441 generic.go:356] "Generic (PLEG): container finished" podID="14741067-19bd-4053-b3c3-7a3211d8222f" containerID="e334be6704695f2121ab89af1343aabba2eadff320f582b096cba797bd74c0cb" exitCode=0 Nov 02 21:17:53.883261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.883240 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" event={"ID":"14741067-19bd-4053-b3c3-7a3211d8222f","Type":"ContainerDied","Data":"e334be6704695f2121ab89af1343aabba2eadff320f582b096cba797bd74c0cb"} Nov 02 21:17:53.885252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.885230 4441 generic.go:356] "Generic (PLEG): container finished" podID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerID="a7e5732fbcc0f13cd292d1c76f4562355bf38c355485f92fbb4414b734ce70a3" exitCode=0 Nov 02 21:17:53.885341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.885307 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" event={"ID":"1f17b360-91e1-492f-bdf0-de861ce4dd43","Type":"ContainerDied","Data":"a7e5732fbcc0f13cd292d1c76f4562355bf38c355485f92fbb4414b734ce70a3"} Nov 02 21:17:53.887719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.887695 4441 generic.go:356] "Generic (PLEG): container finished" podID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerID="c250d66c2b67e732038bfebf2c6c406a19b8c62d43980e9fe3fbbf328f1913e3" exitCode=0 Nov 02 21:17:53.887805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:53.887773 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" event={"ID":"b0d1e821-a731-477d-90ff-53a3fd32c8ab","Type":"ContainerDied","Data":"c250d66c2b67e732038bfebf2c6c406a19b8c62d43980e9fe3fbbf328f1913e3"} Nov 02 21:17:55.925707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:55.924658 4441 generic.go:356] "Generic (PLEG): container finished" podID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerID="b7850e50873aaf62b1cfdad004c4b07ae694d1c536e30fe889f695ddc7140dda" exitCode=0 Nov 02 21:17:55.925707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:17:55.924787 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" event={"ID":"f5179f5c-ef2c-4687-8be2-922114d4433a","Type":"ContainerDied","Data":"b7850e50873aaf62b1cfdad004c4b07ae694d1c536e30fe889f695ddc7140dda"} Nov 02 21:18:02.019869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:02.019168 4441 generic.go:356] "Generic (PLEG): container finished" podID="9717bd05-9191-42c8-b22d-d7771695c05f" containerID="6355056ae53cdeef197fe2534c3f6ca324545fd2c81b77ebed566dfb8d293d9c" exitCode=0 Nov 02 21:18:02.019869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:02.019333 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" event={"ID":"9717bd05-9191-42c8-b22d-d7771695c05f","Type":"ContainerDied","Data":"6355056ae53cdeef197fe2534c3f6ca324545fd2c81b77ebed566dfb8d293d9c"} Nov 02 21:18:10.687296 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:18:10.687150 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:18:17.238207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:17.238162 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" event={"ID":"1f17b360-91e1-492f-bdf0-de861ce4dd43","Type":"ContainerStarted","Data":"c0fbff63c5a7c315254c6d886cea9dce16f39ce10f6814912269248362284dc2"} Nov 02 21:18:17.254520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:17.253936 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" event={"ID":"9717bd05-9191-42c8-b22d-d7771695c05f","Type":"ContainerStarted","Data":"93391e4a6db9dcd8aa7b7994ff735b34fac9a3b6d46b003f3cffea9b9ba24222"} Nov 02 21:18:17.260125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:17.260096 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" event={"ID":"a1957155-7229-4c75-a611-b020695ad17f","Type":"ContainerStarted","Data":"89a61f5db44b6f237efcbda313069de6d1c5ae77e5871cde910edb5ce832feae"} Nov 02 21:18:17.356655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:17.356609 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" podStartSLOduration=2.7527683080000003 podStartE2EDuration="48.356590584s" totalImagesPullingTime="45.603822276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:17.355518387 +0000 UTC m=+326.911146749" watchObservedRunningTime="2026-11-02 21:18:17.356590584 +0000 UTC m=+326.912218923" Nov 02 21:18:17.574810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:17.574750 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" podStartSLOduration=2.818646943 podStartE2EDuration="42.574733045s" totalImagesPullingTime="39.756086102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:17.573998587 +0000 UTC m=+327.129626916" watchObservedRunningTime="2026-11-02 21:18:17.574733045 +0000 UTC m=+327.130361383" Nov 02 21:18:17.576254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:17.576206 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" podStartSLOduration=3.342586701 podStartE2EDuration="44.576193566s" totalImagesPullingTime="41.233606865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:17.465617965 +0000 UTC m=+327.021246305" watchObservedRunningTime="2026-11-02 21:18:17.576193566 +0000 UTC m=+327.131821903" Nov 02 21:18:18.266490 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.266461 4441 generic.go:356] "Generic (PLEG): container finished" podID="a1957155-7229-4c75-a611-b020695ad17f" containerID="89a61f5db44b6f237efcbda313069de6d1c5ae77e5871cde910edb5ce832feae" exitCode=0 Nov 02 21:18:18.266866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.266560 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" event={"ID":"a1957155-7229-4c75-a611-b020695ad17f","Type":"ContainerDied","Data":"89a61f5db44b6f237efcbda313069de6d1c5ae77e5871cde910edb5ce832feae"} Nov 02 21:18:18.269018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.268989 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" event={"ID":"7720b08e-742a-4af9-9dd8-deb90a7d084e","Type":"ContainerStarted","Data":"80bfad5b13dd0daefe987fc640a3adb936d664fbf8fc4017092c0a5c52c3bba1"} Nov 02 21:18:18.270826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.270804 4441 generic.go:356] "Generic (PLEG): container finished" podID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerID="1a3d7e8d57645a6ea8962368eeb8a2038c4baf95fc20042381c3ba7a4bc48505" exitCode=0 Nov 02 21:18:18.270905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.270879 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" event={"ID":"231e8844-6b1f-4f79-a951-bf0e2b8a93d9","Type":"ContainerDied","Data":"1a3d7e8d57645a6ea8962368eeb8a2038c4baf95fc20042381c3ba7a4bc48505"} Nov 02 21:18:18.272747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.272726 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" event={"ID":"14741067-19bd-4053-b3c3-7a3211d8222f","Type":"ContainerStarted","Data":"5c5351c18247e8d8d67435ba8870e6469d3dc352b6fff1ee79a953d6102d8faf"} Nov 02 21:18:18.274703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.274684 4441 generic.go:356] "Generic (PLEG): container finished" podID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerID="c0fbff63c5a7c315254c6d886cea9dce16f39ce10f6814912269248362284dc2" exitCode=0 Nov 02 21:18:18.274780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.274755 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" event={"ID":"1f17b360-91e1-492f-bdf0-de861ce4dd43","Type":"ContainerDied","Data":"c0fbff63c5a7c315254c6d886cea9dce16f39ce10f6814912269248362284dc2"} Nov 02 21:18:18.276549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.276530 4441 generic.go:356] "Generic (PLEG): container finished" podID="9717bd05-9191-42c8-b22d-d7771695c05f" containerID="93391e4a6db9dcd8aa7b7994ff735b34fac9a3b6d46b003f3cffea9b9ba24222" exitCode=0 Nov 02 21:18:18.276645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.276589 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" event={"ID":"9717bd05-9191-42c8-b22d-d7771695c05f","Type":"ContainerDied","Data":"93391e4a6db9dcd8aa7b7994ff735b34fac9a3b6d46b003f3cffea9b9ba24222"} Nov 02 21:18:18.278396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.278373 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" event={"ID":"b0d1e821-a731-477d-90ff-53a3fd32c8ab","Type":"ContainerStarted","Data":"f63d80d5883d42625ef576593b6b7971111cf49f50125c6e2c25455ad11b1f11"} Nov 02 21:18:18.280207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.280186 4441 generic.go:356] "Generic (PLEG): container finished" podID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerID="d24f6d6570f0948f33d021ac59f1a3b59c26eb2727330d8626eb825393bfa5e2" exitCode=0 Nov 02 21:18:18.280279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.280259 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" event={"ID":"07f487c3-36be-4358-9696-d1b1ce4e453f","Type":"ContainerDied","Data":"d24f6d6570f0948f33d021ac59f1a3b59c26eb2727330d8626eb825393bfa5e2"} Nov 02 21:18:18.282375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.282357 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" event={"ID":"f304c488-3134-4049-9f5b-39da4ef4a217","Type":"ContainerStarted","Data":"ffcf853bbd9edd3a7d7a81fe55f10973efeaa7cf1d031bbc36b8ea8712f3be5e"} Nov 02 21:18:18.284238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.284219 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" event={"ID":"f5179f5c-ef2c-4687-8be2-922114d4433a","Type":"ContainerStarted","Data":"c24e3d1e921ebd05919595cc11ee8bb6a9ce1b28d24345c8cce4d41209c4589f"} Nov 02 21:18:18.286274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.286256 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" event={"ID":"441de62c-792a-4bb0-ad70-d6ecc84bd5ad","Type":"ContainerStarted","Data":"3339bb43d068e45431cea1964c73a6a80cdee3f3f97c1d661e198e58ce93524a"} Nov 02 21:18:18.592125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:18.592075 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" podStartSLOduration=3.8024519530000003 podStartE2EDuration="43.592062638s" totalImagesPullingTime="39.789610685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:18.590960247 +0000 UTC m=+328.146588599" watchObservedRunningTime="2026-11-02 21:18:18.592062638 +0000 UTC m=+328.147690974" Nov 02 21:18:19.023701 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.023602 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" podStartSLOduration=4.5237176869999995 podStartE2EDuration="44.023585453s" totalImagesPullingTime="39.499867766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:19.02294066 +0000 UTC m=+328.578568996" watchObservedRunningTime="2026-11-02 21:18:19.023585453 +0000 UTC m=+328.579213790" Nov 02 21:18:19.240410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.240357 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" podStartSLOduration=4.173129977 podStartE2EDuration="51.240343684s" totalImagesPullingTime="47.067213707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:19.239949427 +0000 UTC m=+328.795577764" watchObservedRunningTime="2026-11-02 21:18:19.240343684 +0000 UTC m=+328.795972043" Nov 02 21:18:19.292402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.292337 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:18:19.293049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.293015 4441 generic.go:356] "Generic (PLEG): container finished" podID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerID="80bfad5b13dd0daefe987fc640a3adb936d664fbf8fc4017092c0a5c52c3bba1" exitCode=1 Nov 02 21:18:19.293183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.293097 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" event={"ID":"7720b08e-742a-4af9-9dd8-deb90a7d084e","Type":"ContainerDied","Data":"80bfad5b13dd0daefe987fc640a3adb936d664fbf8fc4017092c0a5c52c3bba1"} Nov 02 21:18:19.295060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.295040 4441 generic.go:356] "Generic (PLEG): container finished" podID="14741067-19bd-4053-b3c3-7a3211d8222f" containerID="5c5351c18247e8d8d67435ba8870e6469d3dc352b6fff1ee79a953d6102d8faf" exitCode=0 Nov 02 21:18:19.295154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.295118 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" event={"ID":"14741067-19bd-4053-b3c3-7a3211d8222f","Type":"ContainerDied","Data":"5c5351c18247e8d8d67435ba8870e6469d3dc352b6fff1ee79a953d6102d8faf"} Nov 02 21:18:19.296658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.296641 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:18:19.297283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.297266 4441 generic.go:356] "Generic (PLEG): container finished" podID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerID="f63d80d5883d42625ef576593b6b7971111cf49f50125c6e2c25455ad11b1f11" exitCode=1 Nov 02 21:18:19.297369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.297349 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" event={"ID":"b0d1e821-a731-477d-90ff-53a3fd32c8ab","Type":"ContainerDied","Data":"f63d80d5883d42625ef576593b6b7971111cf49f50125c6e2c25455ad11b1f11"} Nov 02 21:18:19.299182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.299162 4441 generic.go:356] "Generic (PLEG): container finished" podID="f304c488-3134-4049-9f5b-39da4ef4a217" containerID="ffcf853bbd9edd3a7d7a81fe55f10973efeaa7cf1d031bbc36b8ea8712f3be5e" exitCode=0 Nov 02 21:18:19.299276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.299186 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" event={"ID":"f304c488-3134-4049-9f5b-39da4ef4a217","Type":"ContainerDied","Data":"ffcf853bbd9edd3a7d7a81fe55f10973efeaa7cf1d031bbc36b8ea8712f3be5e"} Nov 02 21:18:19.301126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.301103 4441 generic.go:356] "Generic (PLEG): container finished" podID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerID="c24e3d1e921ebd05919595cc11ee8bb6a9ce1b28d24345c8cce4d41209c4589f" exitCode=0 Nov 02 21:18:19.301215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.301126 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" event={"ID":"f5179f5c-ef2c-4687-8be2-922114d4433a","Type":"ContainerDied","Data":"c24e3d1e921ebd05919595cc11ee8bb6a9ce1b28d24345c8cce4d41209c4589f"} Nov 02 21:18:19.303016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.303000 4441 generic.go:356] "Generic (PLEG): container finished" podID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerID="3339bb43d068e45431cea1964c73a6a80cdee3f3f97c1d661e198e58ce93524a" exitCode=0 Nov 02 21:18:19.303091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.303064 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" event={"ID":"441de62c-792a-4bb0-ad70-d6ecc84bd5ad","Type":"ContainerDied","Data":"3339bb43d068e45431cea1964c73a6a80cdee3f3f97c1d661e198e58ce93524a"} Nov 02 21:18:19.349009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.348922 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" podStartSLOduration=4.570514147 podStartE2EDuration="51.348902676s" totalImagesPullingTime="46.778388529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:19.348566373 +0000 UTC m=+328.904194710" watchObservedRunningTime="2026-11-02 21:18:19.348902676 +0000 UTC m=+328.904531012" Nov 02 21:18:19.458111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.458062 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" podStartSLOduration=2.968839481 podStartE2EDuration="48.458048675s" totalImagesPullingTime="45.489209194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:19.456889762 +0000 UTC m=+329.012518100" watchObservedRunningTime="2026-11-02 21:18:19.458048675 +0000 UTC m=+329.013677012" Nov 02 21:18:19.502989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.502518 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:18:19.532463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.532433 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:18:19.533594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.533558 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-kube-api-access-nnf9n\" (UniqueName: \"kubernetes.io/projected/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-kube-api-access-nnf9n\") pod \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " Nov 02 21:18:19.533721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.533602 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-util\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-util\") pod \"a1957155-7229-4c75-a611-b020695ad17f\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " Nov 02 21:18:19.533721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.533631 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-util\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-util\") pod \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " Nov 02 21:18:19.533721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.533670 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-bundle\") pod \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\" (UID: \"231e8844-6b1f-4f79-a951-bf0e2b8a93d9\") " Nov 02 21:18:19.533721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.533712 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1957155-7229-4c75-a611-b020695ad17f-kube-api-access-7slcq\" (UniqueName: \"kubernetes.io/projected/a1957155-7229-4c75-a611-b020695ad17f-kube-api-access-7slcq\") pod \"a1957155-7229-4c75-a611-b020695ad17f\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " Nov 02 21:18:19.533944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.533769 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-bundle\") pod \"a1957155-7229-4c75-a611-b020695ad17f\" (UID: \"a1957155-7229-4c75-a611-b020695ad17f\") " Nov 02 21:18:19.534401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.534378 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-bundle" pod "a1957155-7229-4c75-a611-b020695ad17f" (UID: "a1957155-7229-4c75-a611-b020695ad17f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.535590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.535348 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-bundle" pod "231e8844-6b1f-4f79-a951-bf0e2b8a93d9" (UID: "231e8844-6b1f-4f79-a951-bf0e2b8a93d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.538890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.538870 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-kube-api-access-nnf9n" pod "231e8844-6b1f-4f79-a951-bf0e2b8a93d9" (UID: "231e8844-6b1f-4f79-a951-bf0e2b8a93d9"). InnerVolumeSpecName "kube-api-access-nnf9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:19.539730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.539708 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-util" pod "a1957155-7229-4c75-a611-b020695ad17f" (UID: "a1957155-7229-4c75-a611-b020695ad17f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.541049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.541014 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1957155-7229-4c75-a611-b020695ad17f-kube-api-access-7slcq" pod "a1957155-7229-4c75-a611-b020695ad17f" (UID: "a1957155-7229-4c75-a611-b020695ad17f"). InnerVolumeSpecName "kube-api-access-7slcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:19.543022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.542913 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-util" pod "231e8844-6b1f-4f79-a951-bf0e2b8a93d9" (UID: "231e8844-6b1f-4f79-a951-bf0e2b8a93d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.565517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.565499 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:18:19.569845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.569823 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:18:19.572287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.572268 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:18:19.575458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.575441 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:18:19.634978 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.634944 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-bundle\") pod \"9717bd05-9191-42c8-b22d-d7771695c05f\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " Nov 02 21:18:19.635075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.634986 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-util\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-util\") pod \"9717bd05-9191-42c8-b22d-d7771695c05f\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " Nov 02 21:18:19.635075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635011 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-bundle\") pod \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " Nov 02 21:18:19.635075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635031 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-kube-api-access-cm2fx\" (UniqueName: \"kubernetes.io/projected/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-kube-api-access-cm2fx\") pod \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " Nov 02 21:18:19.635075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635049 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-bundle\") pod \"07f487c3-36be-4358-9696-d1b1ce4e453f\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " Nov 02 21:18:19.635285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635076 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-util\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-util\") pod \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\" (UID: \"441de62c-792a-4bb0-ad70-d6ecc84bd5ad\") " Nov 02 21:18:19.635285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635109 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f17b360-91e1-492f-bdf0-de861ce4dd43-kube-api-access-6g8md\" (UniqueName: \"kubernetes.io/projected/1f17b360-91e1-492f-bdf0-de861ce4dd43-kube-api-access-6g8md\") pod \"1f17b360-91e1-492f-bdf0-de861ce4dd43\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " Nov 02 21:18:19.635285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635137 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f487c3-36be-4358-9696-d1b1ce4e453f-kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/07f487c3-36be-4358-9696-d1b1ce4e453f-kube-api-access-zmm4c\") pod \"07f487c3-36be-4358-9696-d1b1ce4e453f\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " Nov 02 21:18:19.635285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635165 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9717bd05-9191-42c8-b22d-d7771695c05f-kube-api-access-ptgw6\" (UniqueName: \"kubernetes.io/projected/9717bd05-9191-42c8-b22d-d7771695c05f-kube-api-access-ptgw6\") pod \"9717bd05-9191-42c8-b22d-d7771695c05f\" (UID: \"9717bd05-9191-42c8-b22d-d7771695c05f\") " Nov 02 21:18:19.635285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635208 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-util\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-util\") pod \"07f487c3-36be-4358-9696-d1b1ce4e453f\" (UID: \"07f487c3-36be-4358-9696-d1b1ce4e453f\") " Nov 02 21:18:19.635285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635269 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-bundle\") pod \"1f17b360-91e1-492f-bdf0-de861ce4dd43\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " Nov 02 21:18:19.635574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635314 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-util\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-util\") pod \"1f17b360-91e1-492f-bdf0-de861ce4dd43\" (UID: \"1f17b360-91e1-492f-bdf0-de861ce4dd43\") " Nov 02 21:18:19.635574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635389 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-bundle" pod "9717bd05-9191-42c8-b22d-d7771695c05f" (UID: "9717bd05-9191-42c8-b22d-d7771695c05f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.635679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635638 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-bundle" pod "07f487c3-36be-4358-9696-d1b1ce4e453f" (UID: "07f487c3-36be-4358-9696-d1b1ce4e453f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.635771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635749 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.635828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635776 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.635828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635791 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnf9n\" (UniqueName: \"kubernetes.io/projected/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-kube-api-access-nnf9n\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.635828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635806 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.635828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635818 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.636085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635831 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231e8844-6b1f-4f79-a951-bf0e2b8a93d9-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.636085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635843 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7slcq\" (UniqueName: \"kubernetes.io/projected/a1957155-7229-4c75-a611-b020695ad17f-kube-api-access-7slcq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.636085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635856 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1957155-7229-4c75-a611-b020695ad17f-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.636085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.635909 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-bundle" pod "441de62c-792a-4bb0-ad70-d6ecc84bd5ad" (UID: "441de62c-792a-4bb0-ad70-d6ecc84bd5ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.636720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.636675 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-bundle" pod "1f17b360-91e1-492f-bdf0-de861ce4dd43" (UID: "1f17b360-91e1-492f-bdf0-de861ce4dd43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.638029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.637952 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-kube-api-access-cm2fx" pod "441de62c-792a-4bb0-ad70-d6ecc84bd5ad" (UID: "441de62c-792a-4bb0-ad70-d6ecc84bd5ad"). InnerVolumeSpecName "kube-api-access-cm2fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:19.638327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.638300 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9717bd05-9191-42c8-b22d-d7771695c05f-kube-api-access-ptgw6" pod "9717bd05-9191-42c8-b22d-d7771695c05f" (UID: "9717bd05-9191-42c8-b22d-d7771695c05f"). InnerVolumeSpecName "kube-api-access-ptgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:19.638423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.638324 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f17b360-91e1-492f-bdf0-de861ce4dd43-kube-api-access-6g8md" pod "1f17b360-91e1-492f-bdf0-de861ce4dd43" (UID: "1f17b360-91e1-492f-bdf0-de861ce4dd43"). InnerVolumeSpecName "kube-api-access-6g8md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:19.638809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.638781 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f487c3-36be-4358-9696-d1b1ce4e453f-kube-api-access-zmm4c" pod "07f487c3-36be-4358-9696-d1b1ce4e453f" (UID: "07f487c3-36be-4358-9696-d1b1ce4e453f"). InnerVolumeSpecName "kube-api-access-zmm4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:19.643182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.643160 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-util" pod "441de62c-792a-4bb0-ad70-d6ecc84bd5ad" (UID: "441de62c-792a-4bb0-ad70-d6ecc84bd5ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.643395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.643378 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-util" pod "9717bd05-9191-42c8-b22d-d7771695c05f" (UID: "9717bd05-9191-42c8-b22d-d7771695c05f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.643817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.643796 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-util" pod "07f487c3-36be-4358-9696-d1b1ce4e453f" (UID: "07f487c3-36be-4358-9696-d1b1ce4e453f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.644030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.644010 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-util" pod "1f17b360-91e1-492f-bdf0-de861ce4dd43" (UID: "1f17b360-91e1-492f-bdf0-de861ce4dd43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:19.736500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736475 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f487c3-36be-4358-9696-d1b1ce4e453f-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736502 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736512 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f17b360-91e1-492f-bdf0-de861ce4dd43-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736519 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9717bd05-9191-42c8-b22d-d7771695c05f-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736529 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736537 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2fx\" (UniqueName: \"kubernetes.io/projected/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-kube-api-access-cm2fx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736546 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/441de62c-792a-4bb0-ad70-d6ecc84bd5ad-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736553 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8md\" (UniqueName: \"kubernetes.io/projected/1f17b360-91e1-492f-bdf0-de861ce4dd43-kube-api-access-6g8md\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736561 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/07f487c3-36be-4358-9696-d1b1ce4e453f-kube-api-access-zmm4c\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:19.736582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:19.736570 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptgw6\" (UniqueName: \"kubernetes.io/projected/9717bd05-9191-42c8-b22d-d7771695c05f-kube-api-access-ptgw6\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.309620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.309584 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" Nov 02 21:18:20.309620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.309605 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p95fs" event={"ID":"231e8844-6b1f-4f79-a951-bf0e2b8a93d9","Type":"ContainerDied","Data":"6276357fc99158b0c1708ca5f55f97e4a651637c186fde25f924e88251ead9e2"} Nov 02 21:18:20.310144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.309651 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6276357fc99158b0c1708ca5f55f97e4a651637c186fde25f924e88251ead9e2" Nov 02 21:18:20.311639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.311613 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" event={"ID":"1f17b360-91e1-492f-bdf0-de861ce4dd43","Type":"ContainerDied","Data":"724481a8b7e821feb1469a7978d78b5f1dc95157363b245180bbbb4a62122439"} Nov 02 21:18:20.311639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.311629 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7147hsn" Nov 02 21:18:20.311831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.311652 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="724481a8b7e821feb1469a7978d78b5f1dc95157363b245180bbbb4a62122439" Nov 02 21:18:20.313508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.313489 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" Nov 02 21:18:20.313508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.313495 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8n8h" event={"ID":"9717bd05-9191-42c8-b22d-d7771695c05f","Type":"ContainerDied","Data":"cf871b1adc688d2c3e6fa66f35fa1acc275df48410326146310eeb7c1f43da2c"} Nov 02 21:18:20.313640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.313528 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf871b1adc688d2c3e6fa66f35fa1acc275df48410326146310eeb7c1f43da2c" Nov 02 21:18:20.315419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.315402 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" event={"ID":"07f487c3-36be-4358-9696-d1b1ce4e453f","Type":"ContainerDied","Data":"70f1f1091fd6d5cd34d58bef1cd8823c22a08f2791d945fb5b0b0eb1f3f3b6df"} Nov 02 21:18:20.315508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.315419 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zzxs5" Nov 02 21:18:20.315508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.315434 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f1f1091fd6d5cd34d58bef1cd8823c22a08f2791d945fb5b0b0eb1f3f3b6df" Nov 02 21:18:20.322066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.321804 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" event={"ID":"441de62c-792a-4bb0-ad70-d6ecc84bd5ad","Type":"ContainerDied","Data":"35202b31c48589bc1606656f12c0be8c308c92c705053ddfed13e087be4b0da7"} Nov 02 21:18:20.322066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.321840 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35202b31c48589bc1606656f12c0be8c308c92c705053ddfed13e087be4b0da7" Nov 02 21:18:20.322066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.321907 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdqqqr" Nov 02 21:18:20.324521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.324444 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" Nov 02 21:18:20.324521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.324479 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ks625" event={"ID":"a1957155-7229-4c75-a611-b020695ad17f","Type":"ContainerDied","Data":"0780c37ea5d1afd460338edb657c34229a6439a00354512acfaa5c7cfcf0629b"} Nov 02 21:18:20.324521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.324517 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0780c37ea5d1afd460338edb657c34229a6439a00354512acfaa5c7cfcf0629b" Nov 02 21:18:20.484414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.484395 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:18:20.487534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.487406 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:18:20.509936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.509912 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:18:20.533149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.533125 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:18:20.535155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.535101 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:18:20.542487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542460 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5179f5c-ef2c-4687-8be2-922114d4433a-kube-api-access-jn9jw\" (UniqueName: \"kubernetes.io/projected/f5179f5c-ef2c-4687-8be2-922114d4433a-kube-api-access-jn9jw\") pod \"f5179f5c-ef2c-4687-8be2-922114d4433a\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " Nov 02 21:18:20.542624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542517 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-bundle\") pod \"7720b08e-742a-4af9-9dd8-deb90a7d084e\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " Nov 02 21:18:20.542624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542559 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7720b08e-742a-4af9-9dd8-deb90a7d084e-kube-api-access-kw8vz\" (UniqueName: \"kubernetes.io/projected/7720b08e-742a-4af9-9dd8-deb90a7d084e-kube-api-access-kw8vz\") pod \"7720b08e-742a-4af9-9dd8-deb90a7d084e\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " Nov 02 21:18:20.542624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542583 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-util\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-util\") pod \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " Nov 02 21:18:20.542871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542632 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-util\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-util\") pod \"7720b08e-742a-4af9-9dd8-deb90a7d084e\" (UID: \"7720b08e-742a-4af9-9dd8-deb90a7d084e\") " Nov 02 21:18:20.542871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542667 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-bundle\") pod \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " Nov 02 21:18:20.542871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542727 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-bundle\") pod \"f5179f5c-ef2c-4687-8be2-922114d4433a\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " Nov 02 21:18:20.542871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542753 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-util\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-util\") pod \"f5179f5c-ef2c-4687-8be2-922114d4433a\" (UID: \"f5179f5c-ef2c-4687-8be2-922114d4433a\") " Nov 02 21:18:20.542871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.542791 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d1e821-a731-477d-90ff-53a3fd32c8ab-kube-api-access-bzhwk\" (UniqueName: \"kubernetes.io/projected/b0d1e821-a731-477d-90ff-53a3fd32c8ab-kube-api-access-bzhwk\") pod \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\" (UID: \"b0d1e821-a731-477d-90ff-53a3fd32c8ab\") " Nov 02 21:18:20.543844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.543777 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-bundle" pod "f5179f5c-ef2c-4687-8be2-922114d4433a" (UID: "f5179f5c-ef2c-4687-8be2-922114d4433a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.544732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.544711 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-bundle" pod "b0d1e821-a731-477d-90ff-53a3fd32c8ab" (UID: "b0d1e821-a731-477d-90ff-53a3fd32c8ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.545344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.545320 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5179f5c-ef2c-4687-8be2-922114d4433a-kube-api-access-jn9jw" pod "f5179f5c-ef2c-4687-8be2-922114d4433a" (UID: "f5179f5c-ef2c-4687-8be2-922114d4433a"). InnerVolumeSpecName "kube-api-access-jn9jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:20.545894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.545864 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d1e821-a731-477d-90ff-53a3fd32c8ab-kube-api-access-bzhwk" pod "b0d1e821-a731-477d-90ff-53a3fd32c8ab" (UID: "b0d1e821-a731-477d-90ff-53a3fd32c8ab"). InnerVolumeSpecName "kube-api-access-bzhwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:20.546059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.546040 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-bundle" pod "7720b08e-742a-4af9-9dd8-deb90a7d084e" (UID: "7720b08e-742a-4af9-9dd8-deb90a7d084e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.547076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.547053 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7720b08e-742a-4af9-9dd8-deb90a7d084e-kube-api-access-kw8vz" pod "7720b08e-742a-4af9-9dd8-deb90a7d084e" (UID: "7720b08e-742a-4af9-9dd8-deb90a7d084e"). InnerVolumeSpecName "kube-api-access-kw8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:20.548818 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.548788 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:18:20.550301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.549520 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-util" pod "f5179f5c-ef2c-4687-8be2-922114d4433a" (UID: "f5179f5c-ef2c-4687-8be2-922114d4433a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.550301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.550180 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-util" pod "7720b08e-742a-4af9-9dd8-deb90a7d084e" (UID: "7720b08e-742a-4af9-9dd8-deb90a7d084e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.552044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.552025 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:18:20.552341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.552319 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-util" pod "b0d1e821-a731-477d-90ff-53a3fd32c8ab" (UID: "b0d1e821-a731-477d-90ff-53a3fd32c8ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.644013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.643983 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f304c488-3134-4049-9f5b-39da4ef4a217-kube-api-access-dhr26\" (UniqueName: \"kubernetes.io/projected/f304c488-3134-4049-9f5b-39da4ef4a217-kube-api-access-dhr26\") pod \"f304c488-3134-4049-9f5b-39da4ef4a217\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " Nov 02 21:18:20.644168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644023 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14741067-19bd-4053-b3c3-7a3211d8222f-kube-api-access-xkkmg\" (UniqueName: \"kubernetes.io/projected/14741067-19bd-4053-b3c3-7a3211d8222f-kube-api-access-xkkmg\") pod \"14741067-19bd-4053-b3c3-7a3211d8222f\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " Nov 02 21:18:20.644168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644065 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-util\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-util\") pod \"14741067-19bd-4053-b3c3-7a3211d8222f\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " Nov 02 21:18:20.644168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644119 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-util\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-util\") pod \"f304c488-3134-4049-9f5b-39da4ef4a217\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " Nov 02 21:18:20.644168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644146 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-bundle\") pod \"14741067-19bd-4053-b3c3-7a3211d8222f\" (UID: \"14741067-19bd-4053-b3c3-7a3211d8222f\") " Nov 02 21:18:20.644396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644185 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-bundle\") pod \"f304c488-3134-4049-9f5b-39da4ef4a217\" (UID: \"f304c488-3134-4049-9f5b-39da4ef4a217\") " Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644640 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644667 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644679 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644691 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5179f5c-ef2c-4687-8be2-922114d4433a-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644703 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzhwk\" (UniqueName: \"kubernetes.io/projected/b0d1e821-a731-477d-90ff-53a3fd32c8ab-kube-api-access-bzhwk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644714 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9jw\" (UniqueName: \"kubernetes.io/projected/f5179f5c-ef2c-4687-8be2-922114d4433a-kube-api-access-jn9jw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644723 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7720b08e-742a-4af9-9dd8-deb90a7d084e-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644739 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw8vz\" (UniqueName: \"kubernetes.io/projected/7720b08e-742a-4af9-9dd8-deb90a7d084e-kube-api-access-kw8vz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.644924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.644749 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0d1e821-a731-477d-90ff-53a3fd32c8ab-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.645396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.645183 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-bundle" pod "f304c488-3134-4049-9f5b-39da4ef4a217" (UID: "f304c488-3134-4049-9f5b-39da4ef4a217"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.646210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.646174 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-bundle" pod "14741067-19bd-4053-b3c3-7a3211d8222f" (UID: "14741067-19bd-4053-b3c3-7a3211d8222f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.646846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.646591 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14741067-19bd-4053-b3c3-7a3211d8222f-kube-api-access-xkkmg" pod "14741067-19bd-4053-b3c3-7a3211d8222f" (UID: "14741067-19bd-4053-b3c3-7a3211d8222f"). InnerVolumeSpecName "kube-api-access-xkkmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:20.647621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.647595 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f304c488-3134-4049-9f5b-39da4ef4a217-kube-api-access-dhr26" pod "f304c488-3134-4049-9f5b-39da4ef4a217" (UID: "f304c488-3134-4049-9f5b-39da4ef4a217"). InnerVolumeSpecName "kube-api-access-dhr26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:20.652428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.652403 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-util" pod "f304c488-3134-4049-9f5b-39da4ef4a217" (UID: "f304c488-3134-4049-9f5b-39da4ef4a217"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.654779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.654759 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-util" pod "14741067-19bd-4053-b3c3-7a3211d8222f" (UID: "14741067-19bd-4053-b3c3-7a3211d8222f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:20.745138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.745113 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.745138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.745134 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.745265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.745143 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f304c488-3134-4049-9f5b-39da4ef4a217-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.745265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.745151 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhr26\" (UniqueName: \"kubernetes.io/projected/f304c488-3134-4049-9f5b-39da4ef4a217-kube-api-access-dhr26\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.745265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.745161 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkkmg\" (UniqueName: \"kubernetes.io/projected/14741067-19bd-4053-b3c3-7a3211d8222f-kube-api-access-xkkmg\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:20.745265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:20.745169 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14741067-19bd-4053-b3c3-7a3211d8222f-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:21.329740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.329712 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:18:21.330483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.330458 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" Nov 02 21:18:21.330565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.330457 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6" event={"ID":"7720b08e-742a-4af9-9dd8-deb90a7d084e","Type":"ContainerDied","Data":"8fac0255710ac0047b2e5adf87e8651a9833f3b831845d49c1ab44f9a3e6f9c5"} Nov 02 21:18:21.330648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.330628 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fac0255710ac0047b2e5adf87e8651a9833f3b831845d49c1ab44f9a3e6f9c5" Nov 02 21:18:21.332404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.332379 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" event={"ID":"14741067-19bd-4053-b3c3-7a3211d8222f","Type":"ContainerDied","Data":"94abb758b489ec08c5b208d754e1ad07160e4baa8f20735a07b795b663dd6b25"} Nov 02 21:18:21.332404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.332394 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amqxrh" Nov 02 21:18:21.332589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.332442 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94abb758b489ec08c5b208d754e1ad07160e4baa8f20735a07b795b663dd6b25" Nov 02 21:18:21.333981 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.333949 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:18:21.334692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.334670 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" event={"ID":"b0d1e821-a731-477d-90ff-53a3fd32c8ab","Type":"ContainerDied","Data":"4a7431ea9ff7002b4e78b7300fc53e2708aa5de66ab93407d4b792987c45586d"} Nov 02 21:18:21.334692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.334683 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz" Nov 02 21:18:21.334849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.334706 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7431ea9ff7002b4e78b7300fc53e2708aa5de66ab93407d4b792987c45586d" Nov 02 21:18:21.336661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.336639 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" event={"ID":"f304c488-3134-4049-9f5b-39da4ef4a217","Type":"ContainerDied","Data":"310670cef1bb0d656378367e8d345373f3a0a25f07a47a32fbbc84e85f7c5129"} Nov 02 21:18:21.336776 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.336659 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m5n9q" Nov 02 21:18:21.336776 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.336680 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310670cef1bb0d656378367e8d345373f3a0a25f07a47a32fbbc84e85f7c5129" Nov 02 21:18:21.339654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.339637 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" event={"ID":"f5179f5c-ef2c-4687-8be2-922114d4433a","Type":"ContainerDied","Data":"bd4c8abcf7854504d040a9c013fd371d01853aa8d97e2308def1fdce35b04c0e"} Nov 02 21:18:21.339734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.339667 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd4c8abcf7854504d040a9c013fd371d01853aa8d97e2308def1fdce35b04c0e" Nov 02 21:18:21.339734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:21.339698 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crdl6k" Nov 02 21:18:23.664745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:23.664713 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:23.784060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:23.784025 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:28.044659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044627 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk"] Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044875 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044885 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044896 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044901 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044907 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a1957155-7229-4c75-a611-b020695ad17f" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044912 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1957155-7229-4c75-a611-b020695ad17f" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044920 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerName="pull" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044928 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerName="pull" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044937 4441 cpu_manager.go:414] "containerMap: removing container" podUID="9717bd05-9191-42c8-b22d-d7771695c05f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044944 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9717bd05-9191-42c8-b22d-d7771695c05f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044953 4441 cpu_manager.go:414] "containerMap: removing container" podUID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044960 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044980 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f304c488-3134-4049-9f5b-39da4ef4a217" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044985 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f304c488-3134-4049-9f5b-39da4ef4a217" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044991 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.044995 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045004 4441 cpu_manager.go:414] "containerMap: removing container" podUID="14741067-19bd-4053-b3c3-7a3211d8222f" containerName="pull" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045008 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14741067-19bd-4053-b3c3-7a3211d8222f" containerName="pull" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045015 4441 cpu_manager.go:414] "containerMap: removing container" podUID="14741067-19bd-4053-b3c3-7a3211d8222f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045019 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14741067-19bd-4053-b3c3-7a3211d8222f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045027 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f304c488-3134-4049-9f5b-39da4ef4a217" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045032 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f304c488-3134-4049-9f5b-39da4ef4a217" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045038 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a1957155-7229-4c75-a611-b020695ad17f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045043 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1957155-7229-4c75-a611-b020695ad17f" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045048 4441 cpu_manager.go:414] "containerMap: removing container" podUID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045053 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerName="extract" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045060 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045064 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerName="util" Nov 02 21:18:28.045055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045071 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a1957155-7229-4c75-a611-b020695ad17f" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045075 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1957155-7229-4c75-a611-b020695ad17f" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045082 4441 cpu_manager.go:414] "containerMap: removing container" podUID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045086 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045092 4441 cpu_manager.go:414] "containerMap: removing container" podUID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045096 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045102 4441 cpu_manager.go:414] "containerMap: removing container" podUID="9717bd05-9191-42c8-b22d-d7771695c05f" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045106 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9717bd05-9191-42c8-b22d-d7771695c05f" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045113 4441 cpu_manager.go:414] "containerMap: removing container" podUID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045116 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045121 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045125 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045131 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f304c488-3134-4049-9f5b-39da4ef4a217" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045135 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f304c488-3134-4049-9f5b-39da4ef4a217" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045140 4441 cpu_manager.go:414] "containerMap: removing container" podUID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045144 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045151 4441 cpu_manager.go:414] "containerMap: removing container" podUID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045154 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045160 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045163 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045171 4441 cpu_manager.go:414] "containerMap: removing container" podUID="9717bd05-9191-42c8-b22d-d7771695c05f" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045175 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9717bd05-9191-42c8-b22d-d7771695c05f" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045181 4441 cpu_manager.go:414] "containerMap: removing container" podUID="14741067-19bd-4053-b3c3-7a3211d8222f" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045184 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14741067-19bd-4053-b3c3-7a3211d8222f" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045190 4441 cpu_manager.go:414] "containerMap: removing container" podUID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045194 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045200 4441 cpu_manager.go:414] "containerMap: removing container" podUID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045203 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045208 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045212 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045220 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045224 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerName="util" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045229 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045233 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045243 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045247 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045253 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045257 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerName="pull" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045297 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f17b360-91e1-492f-bdf0-de861ce4dd43" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045306 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="9717bd05-9191-42c8-b22d-d7771695c05f" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045312 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f487c3-36be-4358-9696-d1b1ce4e453f" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045318 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7720b08e-742a-4af9-9dd8-deb90a7d084e" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045322 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f304c488-3134-4049-9f5b-39da4ef4a217" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045327 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="14741067-19bd-4053-b3c3-7a3211d8222f" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045332 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1957155-7229-4c75-a611-b020695ad17f" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045337 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="231e8844-6b1f-4f79-a951-bf0e2b8a93d9" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045342 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d1e821-a731-477d-90ff-53a3fd32c8ab" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045347 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5179f5c-ef2c-4687-8be2-922114d4433a" containerName="extract" Nov 02 21:18:28.045895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.045352 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="441de62c-792a-4bb0-ad70-d6ecc84bd5ad" containerName="extract" Nov 02 21:18:28.048186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.048165 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.102835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.102806 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.103043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.102849 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nck8\" (UniqueName: \"kubernetes.io/projected/409228eb-796d-4cf3-8d7b-c259cf986ed3-kube-api-access-9nck8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.103043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.102928 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.146812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.146782 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 21:18:28.146812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.146797 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6bc4z\"" Nov 02 21:18:28.146989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.146881 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 21:18:28.148752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.148735 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk"] Nov 02 21:18:28.203574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.203550 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nck8\" (UniqueName: \"kubernetes.io/projected/409228eb-796d-4cf3-8d7b-c259cf986ed3-kube-api-access-9nck8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.203667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.203578 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.203667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.203642 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.204037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.204018 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.204101 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.204081 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.258371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.258346 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nck8\" (UniqueName: \"kubernetes.io/projected/409228eb-796d-4cf3-8d7b-c259cf986ed3-kube-api-access-9nck8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.360937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.360902 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:28.724048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:28.723961 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk"] Nov 02 21:18:28.731164 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:28.731137 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409228eb_796d_4cf3_8d7b_c259cf986ed3.slice/crio-bd3f5a9cae9798563c481af5e5652d28ed22aa753f293dc8719e9a6f8dbfc4e3 WatchSource:0}: Error finding container bd3f5a9cae9798563c481af5e5652d28ed22aa753f293dc8719e9a6f8dbfc4e3: Status 404 returned error can't find the container with id bd3f5a9cae9798563c481af5e5652d28ed22aa753f293dc8719e9a6f8dbfc4e3 Nov 02 21:18:29.376284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:29.376245 4441 generic.go:356] "Generic (PLEG): container finished" podID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerID="6a4ce32b655224473d52308e5967a5e5eaed03fc72818ff43cf3a9e92f8639eb" exitCode=0 Nov 02 21:18:29.376741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:29.376328 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" event={"ID":"409228eb-796d-4cf3-8d7b-c259cf986ed3","Type":"ContainerDied","Data":"6a4ce32b655224473d52308e5967a5e5eaed03fc72818ff43cf3a9e92f8639eb"} Nov 02 21:18:29.376741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:29.376402 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" event={"ID":"409228eb-796d-4cf3-8d7b-c259cf986ed3","Type":"ContainerStarted","Data":"bd3f5a9cae9798563c481af5e5652d28ed22aa753f293dc8719e9a6f8dbfc4e3"} Nov 02 21:18:30.383257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:30.383224 4441 generic.go:356] "Generic (PLEG): container finished" podID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerID="b3e6dc283b50ef535a4039d680cbb6e65b4f67b97b6bf051a2063d69a8e2d34f" exitCode=0 Nov 02 21:18:30.383625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:30.383267 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" event={"ID":"409228eb-796d-4cf3-8d7b-c259cf986ed3","Type":"ContainerDied","Data":"b3e6dc283b50ef535a4039d680cbb6e65b4f67b97b6bf051a2063d69a8e2d34f"} Nov 02 21:18:31.390046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:31.390007 4441 generic.go:356] "Generic (PLEG): container finished" podID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerID="09a899f25947303a445548149e196b3c4cfd0d78cb4f9201b7bdae6bda2381f6" exitCode=0 Nov 02 21:18:31.390445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:31.390061 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" event={"ID":"409228eb-796d-4cf3-8d7b-c259cf986ed3","Type":"ContainerDied","Data":"09a899f25947303a445548149e196b3c4cfd0d78cb4f9201b7bdae6bda2381f6"} Nov 02 21:18:31.564457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:31.564397 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:31.789708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:31.789623 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:32.512603 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.512578 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:32.535916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.535888 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-bundle\") pod \"409228eb-796d-4cf3-8d7b-c259cf986ed3\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " Nov 02 21:18:32.536060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.535960 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-util\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-util\") pod \"409228eb-796d-4cf3-8d7b-c259cf986ed3\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " Nov 02 21:18:32.536060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.536035 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409228eb-796d-4cf3-8d7b-c259cf986ed3-kube-api-access-9nck8\" (UniqueName: \"kubernetes.io/projected/409228eb-796d-4cf3-8d7b-c259cf986ed3-kube-api-access-9nck8\") pod \"409228eb-796d-4cf3-8d7b-c259cf986ed3\" (UID: \"409228eb-796d-4cf3-8d7b-c259cf986ed3\") " Nov 02 21:18:32.536522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.536489 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-bundle" pod "409228eb-796d-4cf3-8d7b-c259cf986ed3" (UID: "409228eb-796d-4cf3-8d7b-c259cf986ed3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:32.537938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.537911 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409228eb-796d-4cf3-8d7b-c259cf986ed3-kube-api-access-9nck8" pod "409228eb-796d-4cf3-8d7b-c259cf986ed3" (UID: "409228eb-796d-4cf3-8d7b-c259cf986ed3"). InnerVolumeSpecName "kube-api-access-9nck8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:32.542507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.542485 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-util" pod "409228eb-796d-4cf3-8d7b-c259cf986ed3" (UID: "409228eb-796d-4cf3-8d7b-c259cf986ed3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:32.637369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.637342 4441 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-util\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:32.637369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.637368 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nck8\" (UniqueName: \"kubernetes.io/projected/409228eb-796d-4cf3-8d7b-c259cf986ed3-kube-api-access-9nck8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:32.637536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.637379 4441 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409228eb-796d-4cf3-8d7b-c259cf986ed3-bundle\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:32.709568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.709500 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:32.886205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886170 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7"] Nov 02 21:18:32.886456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886445 4441 cpu_manager.go:414] "containerMap: removing container" podUID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerName="util" Nov 02 21:18:32.886500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886458 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerName="util" Nov 02 21:18:32.886500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886468 4441 cpu_manager.go:414] "containerMap: removing container" podUID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerName="extract" Nov 02 21:18:32.886500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886473 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerName="extract" Nov 02 21:18:32.886500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886486 4441 cpu_manager.go:414] "containerMap: removing container" podUID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerName="pull" Nov 02 21:18:32.886500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886490 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerName="pull" Nov 02 21:18:32.886645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.886536 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="409228eb-796d-4cf3-8d7b-c259cf986ed3" containerName="extract" Nov 02 21:18:32.889542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.889527 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:32.940073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940044 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-metrics-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:32.940190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940079 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdff266-eea9-4789-9d90-d252fab9c2bf-tmp\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:32.940190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940100 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpm7\" (UniqueName: \"kubernetes.io/projected/fbdff266-eea9-4789-9d90-d252fab9c2bf-kube-api-access-4vpm7\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:32.940190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940118 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbdff266-eea9-4789-9d90-d252fab9c2bf-manifests\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:32.940190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940184 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 21:18:32.940378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940342 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:32.940446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940412 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 21:18:32.940446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940415 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:32.940563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940445 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 21:18:32.940563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940471 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-mt8kr\"" Nov 02 21:18:32.940563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940482 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 21:18:32.940563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.940555 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 21:18:32.989851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:32.989789 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7"] Nov 02 21:18:33.006279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.006237 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:33.040768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.040745 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.040876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.040778 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-metrics-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.040876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.040794 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdff266-eea9-4789-9d90-d252fab9c2bf-tmp\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.040876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.040818 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpm7\" (UniqueName: \"kubernetes.io/projected/fbdff266-eea9-4789-9d90-d252fab9c2bf-kube-api-access-4vpm7\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.040876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.040844 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbdff266-eea9-4789-9d90-d252fab9c2bf-manifests\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.041060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.040905 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.041307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.041286 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbdff266-eea9-4789-9d90-d252fab9c2bf-manifests\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.043188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.043167 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdff266-eea9-4789-9d90-d252fab9c2bf-tmp\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.043288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.043273 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.043407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.043392 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.043558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.043536 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbdff266-eea9-4789-9d90-d252fab9c2bf-metrics-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.095137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.095118 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpm7\" (UniqueName: \"kubernetes.io/projected/fbdff266-eea9-4789-9d90-d252fab9c2bf-kube-api-access-4vpm7\") pod \"opendatahub-operator-controller-manager-676bcb69b5-ccld7\" (UID: \"fbdff266-eea9-4789-9d90-d252fab9c2bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.152104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.152061 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:33.203766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.203745 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:33.300719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.300679 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:33.405357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.405332 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" Nov 02 21:18:33.405357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.405341 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab5jrk" event={"ID":"409228eb-796d-4cf3-8d7b-c259cf986ed3","Type":"ContainerDied","Data":"bd3f5a9cae9798563c481af5e5652d28ed22aa753f293dc8719e9a6f8dbfc4e3"} Nov 02 21:18:33.405560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.405415 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd3f5a9cae9798563c481af5e5652d28ed22aa753f293dc8719e9a6f8dbfc4e3" Nov 02 21:18:33.427522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.427497 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:33.715249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:33.715212 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7"] Nov 02 21:18:33.722585 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:33.722551 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdff266_eea9_4789_9d90_d252fab9c2bf.slice/crio-168f7b5502a24cbe3ac7d8dec71171982b60455ec3528f6eb40cdee0ed287632 WatchSource:0}: Error finding container 168f7b5502a24cbe3ac7d8dec71171982b60455ec3528f6eb40cdee0ed287632: Status 404 returned error can't find the container with id 168f7b5502a24cbe3ac7d8dec71171982b60455ec3528f6eb40cdee0ed287632 Nov 02 21:18:34.160307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:34.160278 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:34.410565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:34.410487 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" event={"ID":"fbdff266-eea9-4789-9d90-d252fab9c2bf","Type":"ContainerStarted","Data":"168f7b5502a24cbe3ac7d8dec71171982b60455ec3528f6eb40cdee0ed287632"} Nov 02 21:18:36.182606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:36.182570 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:37.289813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.289738 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh"] Nov 02 21:18:37.295942 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.295921 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:37.336586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.336564 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh"] Nov 02 21:18:37.377434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.377406 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95sq\" (UniqueName: \"kubernetes.io/projected/b052f41a-7905-40da-b80f-8d059889ab57-kube-api-access-f95sq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh\" (UID: \"b052f41a-7905-40da-b80f-8d059889ab57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:37.377542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.377478 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b052f41a-7905-40da-b80f-8d059889ab57-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh\" (UID: \"b052f41a-7905-40da-b80f-8d059889ab57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:37.382180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.382160 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:18:37.382274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.382233 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lj7pn\"" Nov 02 21:18:37.382274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.382266 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 21:18:37.430236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.430208 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" event={"ID":"fbdff266-eea9-4789-9d90-d252fab9c2bf","Type":"ContainerStarted","Data":"23db3932c0d44ec9ddc418282a1e7ef4bd3b1c1e9e43bbbe94a02b809ecf5ae7"} Nov 02 21:18:37.478144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.478125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b052f41a-7905-40da-b80f-8d059889ab57-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh\" (UID: \"b052f41a-7905-40da-b80f-8d059889ab57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:37.478295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.478274 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95sq\" (UniqueName: \"kubernetes.io/projected/b052f41a-7905-40da-b80f-8d059889ab57-kube-api-access-f95sq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh\" (UID: \"b052f41a-7905-40da-b80f-8d059889ab57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:37.478599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.478567 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b052f41a-7905-40da-b80f-8d059889ab57-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh\" (UID: \"b052f41a-7905-40da-b80f-8d059889ab57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:37.532330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.532307 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95sq\" (UniqueName: \"kubernetes.io/projected/b052f41a-7905-40da-b80f-8d059889ab57-kube-api-access-f95sq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh\" (UID: \"b052f41a-7905-40da-b80f-8d059889ab57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:37.607613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:37.607579 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" Nov 02 21:18:38.031753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:38.031677 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh"] Nov 02 21:18:38.038773 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:38.038728 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb052f41a_7905_40da_b80f_8d059889ab57.slice/crio-b51b9cfc7cc6b13c54e6966260a30429e77df3add683b0ddf7b7bb2602d4033d WatchSource:0}: Error finding container b51b9cfc7cc6b13c54e6966260a30429e77df3add683b0ddf7b7bb2602d4033d: Status 404 returned error can't find the container with id b51b9cfc7cc6b13c54e6966260a30429e77df3add683b0ddf7b7bb2602d4033d Nov 02 21:18:38.436376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:38.436341 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" event={"ID":"b052f41a-7905-40da-b80f-8d059889ab57","Type":"ContainerStarted","Data":"b51b9cfc7cc6b13c54e6966260a30429e77df3add683b0ddf7b7bb2602d4033d"} Nov 02 21:18:38.437926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:38.437899 4441 generic.go:356] "Generic (PLEG): container finished" podID="fbdff266-eea9-4789-9d90-d252fab9c2bf" containerID="23db3932c0d44ec9ddc418282a1e7ef4bd3b1c1e9e43bbbe94a02b809ecf5ae7" exitCode=0 Nov 02 21:18:38.438069 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:38.437949 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" event={"ID":"fbdff266-eea9-4789-9d90-d252fab9c2bf","Type":"ContainerDied","Data":"23db3932c0d44ec9ddc418282a1e7ef4bd3b1c1e9e43bbbe94a02b809ecf5ae7"} Nov 02 21:18:39.443725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:39.443691 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" event={"ID":"fbdff266-eea9-4789-9d90-d252fab9c2bf","Type":"ContainerStarted","Data":"7fb8c114ba6efb971d27c3c6b65058be81c299667de973c2f4182224f864f617"} Nov 02 21:18:39.444069 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:39.443809 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:39.667418 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:39.667371 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" podStartSLOduration=3.106017646 podStartE2EDuration="7.667356462s" totalImagesPullingTime="4.118272243s" totalInitContainerRuntime="443.066573ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:39.618727789 +0000 UTC m=+349.174356127" watchObservedRunningTime="2026-11-02 21:18:39.667356462 +0000 UTC m=+349.222984799" Nov 02 21:18:39.953437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:39.953366 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:40.617122 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:18:40.617077 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:18:41.204240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:41.204139 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:41.453476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:41.453443 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" event={"ID":"b052f41a-7905-40da-b80f-8d059889ab57","Type":"ContainerStarted","Data":"b900748db6cb591656869d9d67f0160b4d5c8f1220dce8b1594ad70b918de202"} Nov 02 21:18:41.660756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:41.660703 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mbmsh" podStartSLOduration=1.80144218 podStartE2EDuration="4.660690153s" totalImagesPullingTime="2.859247973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:41.659825928 +0000 UTC m=+351.215454264" watchObservedRunningTime="2026-11-02 21:18:41.660690153 +0000 UTC m=+351.216318489" Nov 02 21:18:42.060987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:42.060944 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:42.496990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:42.496896 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:42.719979 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:42.719911 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:42.856269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:42.856224 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:43.545685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.545652 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl"] Nov 02 21:18:43.551894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.551870 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:43.601499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.601471 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl"] Nov 02 21:18:43.626826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.626789 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smf8\" (UniqueName: \"kubernetes.io/projected/ace09442-f99c-432d-8f6a-995c68786fb0-kube-api-access-9smf8\") pod \"cert-manager-webhook-5dbbbf75cc-2k8gl\" (UID: \"ace09442-f99c-432d-8f6a-995c68786fb0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:43.626988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.626909 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ace09442-f99c-432d-8f6a-995c68786fb0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2k8gl\" (UID: \"ace09442-f99c-432d-8f6a-995c68786fb0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:43.646459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.646435 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 21:18:43.646655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.646641 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 21:18:43.646696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.646654 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xplmj\"" Nov 02 21:18:43.728050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.728015 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smf8\" (UniqueName: \"kubernetes.io/projected/ace09442-f99c-432d-8f6a-995c68786fb0-kube-api-access-9smf8\") pod \"cert-manager-webhook-5dbbbf75cc-2k8gl\" (UID: \"ace09442-f99c-432d-8f6a-995c68786fb0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:43.728508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.728063 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ace09442-f99c-432d-8f6a-995c68786fb0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2k8gl\" (UID: \"ace09442-f99c-432d-8f6a-995c68786fb0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:43.782366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.782337 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ace09442-f99c-432d-8f6a-995c68786fb0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2k8gl\" (UID: \"ace09442-f99c-432d-8f6a-995c68786fb0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:43.782555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.782536 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smf8\" (UniqueName: \"kubernetes.io/projected/ace09442-f99c-432d-8f6a-995c68786fb0-kube-api-access-9smf8\") pod \"cert-manager-webhook-5dbbbf75cc-2k8gl\" (UID: \"ace09442-f99c-432d-8f6a-995c68786fb0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:43.874171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:43.874146 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:44.238673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.238593 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl"] Nov 02 21:18:44.245013 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:44.244986 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podace09442_f99c_432d_8f6a_995c68786fb0.slice/crio-7e2f3894d3c801e7f8770261c9c74227fa01a28b51e350dff68ba1b008eda339 WatchSource:0}: Error finding container 7e2f3894d3c801e7f8770261c9c74227fa01a28b51e350dff68ba1b008eda339: Status 404 returned error can't find the container with id 7e2f3894d3c801e7f8770261c9c74227fa01a28b51e350dff68ba1b008eda339 Nov 02 21:18:44.466507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.466474 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" event={"ID":"ace09442-f99c-432d-8f6a-995c68786fb0","Type":"ContainerStarted","Data":"7e2f3894d3c801e7f8770261c9c74227fa01a28b51e350dff68ba1b008eda339"} Nov 02 21:18:44.481658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.481631 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:44.717510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.717478 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8"] Nov 02 21:18:44.724544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.724517 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" Nov 02 21:18:44.727290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.727269 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:44.774497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.774466 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 21:18:44.774497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.774465 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 21:18:44.775307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.775286 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8"] Nov 02 21:18:44.820363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.820335 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w629m\"" Nov 02 21:18:44.837874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.837842 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9dt\" (UniqueName: \"kubernetes.io/projected/73d816d8-936a-4ff8-a6e6-f49cb87788a5-kube-api-access-6b9dt\") pod \"authorino-operator-7b7bb4bc77-g8lw8\" (UID: \"73d816d8-936a-4ff8-a6e6-f49cb87788a5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" Nov 02 21:18:44.939094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.939062 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9dt\" (UniqueName: \"kubernetes.io/projected/73d816d8-936a-4ff8-a6e6-f49cb87788a5-kube-api-access-6b9dt\") pod \"authorino-operator-7b7bb4bc77-g8lw8\" (UID: \"73d816d8-936a-4ff8-a6e6-f49cb87788a5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" Nov 02 21:18:44.993793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:44.993727 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9dt\" (UniqueName: \"kubernetes.io/projected/73d816d8-936a-4ff8-a6e6-f49cb87788a5-kube-api-access-6b9dt\") pod \"authorino-operator-7b7bb4bc77-g8lw8\" (UID: \"73d816d8-936a-4ff8-a6e6-f49cb87788a5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" Nov 02 21:18:45.037888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:45.037853 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" Nov 02 21:18:45.409263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:45.409228 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8"] Nov 02 21:18:45.416412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:45.416385 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d816d8_936a_4ff8_a6e6_f49cb87788a5.slice/crio-4a29ff0e571d88acb372aeac95e8ad6a6cd62176c222a94a218ea62f90899eb5 WatchSource:0}: Error finding container 4a29ff0e571d88acb372aeac95e8ad6a6cd62176c222a94a218ea62f90899eb5: Status 404 returned error can't find the container with id 4a29ff0e571d88acb372aeac95e8ad6a6cd62176c222a94a218ea62f90899eb5 Nov 02 21:18:45.439344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:45.439317 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:45.472400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:45.472374 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" event={"ID":"73d816d8-936a-4ff8-a6e6-f49cb87788a5","Type":"ContainerStarted","Data":"4a29ff0e571d88acb372aeac95e8ad6a6cd62176c222a94a218ea62f90899eb5"} Nov 02 21:18:45.959167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:45.959123 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:47.786599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:47.786559 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:48.492527 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.492481 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" event={"ID":"ace09442-f99c-432d-8f6a-995c68786fb0","Type":"ContainerStarted","Data":"2224eecdf20f459fefc2311b5fa339b388ccbf32ceda0045454b7acbbe0d38e8"} Nov 02 21:18:48.492686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.492663 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:48.493593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.493570 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8"] Nov 02 21:18:48.498205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.498184 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" Nov 02 21:18:48.552444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.552413 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8"] Nov 02 21:18:48.572997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.572948 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwmq\" (UniqueName: \"kubernetes.io/projected/b0d2478d-1641-4f16-8abb-c044269fc79b-kube-api-access-tzwmq\") pod \"limitador-operator-controller-manager-5778bb4449-wz5w8\" (UID: \"b0d2478d-1641-4f16-8abb-c044269fc79b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" Nov 02 21:18:48.589410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.589384 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dnjx7\"" Nov 02 21:18:48.601429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.601374 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" podStartSLOduration=1.469328043 podStartE2EDuration="5.601355324s" totalImagesPullingTime="4.132027281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:48.599490835 +0000 UTC m=+358.155119173" watchObservedRunningTime="2026-11-02 21:18:48.601355324 +0000 UTC m=+358.156983662" Nov 02 21:18:48.674187 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.674112 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwmq\" (UniqueName: \"kubernetes.io/projected/b0d2478d-1641-4f16-8abb-c044269fc79b-kube-api-access-tzwmq\") pod \"limitador-operator-controller-manager-5778bb4449-wz5w8\" (UID: \"b0d2478d-1641-4f16-8abb-c044269fc79b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" Nov 02 21:18:48.728760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.728729 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwmq\" (UniqueName: \"kubernetes.io/projected/b0d2478d-1641-4f16-8abb-c044269fc79b-kube-api-access-tzwmq\") pod \"limitador-operator-controller-manager-5778bb4449-wz5w8\" (UID: \"b0d2478d-1641-4f16-8abb-c044269fc79b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" Nov 02 21:18:48.813357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:48.813315 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" Nov 02 21:18:49.184159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:49.184134 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8"] Nov 02 21:18:49.190882 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:49.190842 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d2478d_1641_4f16_8abb_c044269fc79b.slice/crio-d18fbf685879d76599533bb364d4d87295dcd859560941956d8b08c11a07c345 WatchSource:0}: Error finding container d18fbf685879d76599533bb364d4d87295dcd859560941956d8b08c11a07c345: Status 404 returned error can't find the container with id d18fbf685879d76599533bb364d4d87295dcd859560941956d8b08c11a07c345 Nov 02 21:18:49.500234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:49.500154 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" event={"ID":"b0d2478d-1641-4f16-8abb-c044269fc79b","Type":"ContainerStarted","Data":"d18fbf685879d76599533bb364d4d87295dcd859560941956d8b08c11a07c345"} Nov 02 21:18:49.502137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:49.502108 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" event={"ID":"73d816d8-936a-4ff8-a6e6-f49cb87788a5","Type":"ContainerStarted","Data":"11fbc482705fe17e09eef397a0542574b50c35bb68dc386f75fe4d512bc36f34"} Nov 02 21:18:49.555264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:49.555237 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:49.610121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:49.610036 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" podStartSLOduration=1.901235762 podStartE2EDuration="5.610020767s" totalImagesPullingTime="3.708785005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:49.609365205 +0000 UTC m=+359.164993556" watchObservedRunningTime="2026-11-02 21:18:49.610020767 +0000 UTC m=+359.165649104" Nov 02 21:18:50.142010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:50.141962 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:50.450681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:50.450604 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-ccld7" Nov 02 21:18:50.506145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:50.506120 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" Nov 02 21:18:50.744494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:50.744421 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:51.233243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:51.233211 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:52.342864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:52.342829 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:52.463978 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:52.463947 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:52.516421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:52.516390 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" event={"ID":"b0d2478d-1641-4f16-8abb-c044269fc79b","Type":"ContainerStarted","Data":"abb8b2bd34411c3b3cf8efdb08a1d06312c3a56fd93a8002e792e4237b093c2a"} Nov 02 21:18:52.516560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:52.516464 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" Nov 02 21:18:52.644028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:52.643940 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:52.670615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:52.670562 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" podStartSLOduration=2.029874336 podStartE2EDuration="4.670549689s" totalImagesPullingTime="2.640675353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:52.669643749 +0000 UTC m=+362.225272080" watchObservedRunningTime="2026-11-02 21:18:52.670549689 +0000 UTC m=+362.226178026" Nov 02 21:18:52.858377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:52.858342 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:53.209241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:53.209213 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:54.504778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:54.504747 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" Nov 02 21:18:54.829994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:54.829953 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg"] Nov 02 21:18:54.833308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:54.833289 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:54.884012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:54.883958 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v8rj6\"" Nov 02 21:18:54.922309 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:54.922277 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqddd\" (UniqueName: \"kubernetes.io/projected/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-kube-api-access-xqddd\") pod \"kuadrant-operator-controller-manager-678d87766f-627xg\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:54.922426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:54.922330 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-627xg\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:54.933132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:54.933101 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg"] Nov 02 21:18:55.023270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:55.023238 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqddd\" (UniqueName: \"kubernetes.io/projected/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-kube-api-access-xqddd\") pod \"kuadrant-operator-controller-manager-678d87766f-627xg\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:55.023390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:55.023282 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-627xg\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:55.023740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:55.023722 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-627xg\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:55.085081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:55.085027 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqddd\" (UniqueName: \"kubernetes.io/projected/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-kube-api-access-xqddd\") pod \"kuadrant-operator-controller-manager-678d87766f-627xg\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:55.144984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:55.144944 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:18:55.566733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:55.566702 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg"] Nov 02 21:18:55.573775 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:55.573749 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580cef2d_3c2b_4b12_bfe5_d82f44cfaba5.slice/crio-20ff36fdbfefca7a8b94a14087b3de4df05071d62dcf95ac357e8f4115efbfa8 WatchSource:0}: Error finding container 20ff36fdbfefca7a8b94a14087b3de4df05071d62dcf95ac357e8f4115efbfa8: Status 404 returned error can't find the container with id 20ff36fdbfefca7a8b94a14087b3de4df05071d62dcf95ac357e8f4115efbfa8 Nov 02 21:18:55.768963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:55.768930 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:56.537559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:56.537525 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" event={"ID":"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5","Type":"ContainerStarted","Data":"20ff36fdbfefca7a8b94a14087b3de4df05071d62dcf95ac357e8f4115efbfa8"} Nov 02 21:18:57.198282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.198249 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5"] Nov 02 21:18:57.202895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.202874 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" Nov 02 21:18:57.211316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.211290 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:57.256055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.256033 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 21:18:57.303408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.303378 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-d4bt7\"" Nov 02 21:18:57.305373 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.305353 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5"] Nov 02 21:18:57.338359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.338332 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lpk\" (UniqueName: \"kubernetes.io/projected/2bff1a62-9aa4-478c-a9a9-f95e1c1fd094-kube-api-access-n5lpk\") pod \"dns-operator-controller-manager-8557cbdb84-4zzv5\" (UID: \"2bff1a62-9aa4-478c-a9a9-f95e1c1fd094\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" Nov 02 21:18:57.439170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.439143 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lpk\" (UniqueName: \"kubernetes.io/projected/2bff1a62-9aa4-478c-a9a9-f95e1c1fd094-kube-api-access-n5lpk\") pod \"dns-operator-controller-manager-8557cbdb84-4zzv5\" (UID: \"2bff1a62-9aa4-478c-a9a9-f95e1c1fd094\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" Nov 02 21:18:57.499583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.499527 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lpk\" (UniqueName: \"kubernetes.io/projected/2bff1a62-9aa4-478c-a9a9-f95e1c1fd094-kube-api-access-n5lpk\") pod \"dns-operator-controller-manager-8557cbdb84-4zzv5\" (UID: \"2bff1a62-9aa4-478c-a9a9-f95e1c1fd094\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" Nov 02 21:18:57.516327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.516298 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" Nov 02 21:18:57.894173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:57.894138 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5"] Nov 02 21:18:57.900334 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:18:57.900308 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bff1a62_9aa4_478c_a9a9_f95e1c1fd094.slice/crio-6dec40ef7659e9bdcd58b414bc4eff14904e5c81cc5e7274dad3e2f630cef3b2 WatchSource:0}: Error finding container 6dec40ef7659e9bdcd58b414bc4eff14904e5c81cc5e7274dad3e2f630cef3b2: Status 404 returned error can't find the container with id 6dec40ef7659e9bdcd58b414bc4eff14904e5c81cc5e7274dad3e2f630cef3b2 Nov 02 21:18:58.553935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:58.553893 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" event={"ID":"2bff1a62-9aa4-478c-a9a9-f95e1c1fd094","Type":"ContainerStarted","Data":"6dec40ef7659e9bdcd58b414bc4eff14904e5c81cc5e7274dad3e2f630cef3b2"} Nov 02 21:18:58.765019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:58.764956 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:58.893210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:58.893130 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:59.156532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:59.156453 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:59.290606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:59.290557 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:59.869755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:59.869716 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:59.999627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:18:59.999592 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:00.353120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:00.353081 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:00.496398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:00.496367 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:01.513634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:01.513604 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8lw8" Nov 02 21:19:01.571608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:01.571563 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" event={"ID":"2bff1a62-9aa4-478c-a9a9-f95e1c1fd094","Type":"ContainerStarted","Data":"f51e98a7f2da93367fd700af5188eda1a9e48e96d0625003a009a2102b0a9c02"} Nov 02 21:19:01.571795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:01.571716 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" Nov 02 21:19:01.573713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:01.573675 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" event={"ID":"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5","Type":"ContainerStarted","Data":"ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71"} Nov 02 21:19:01.573835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:01.573739 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:19:01.742071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:01.742020 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" podStartSLOduration=2.710790968 podStartE2EDuration="7.742006251s" totalImagesPullingTime="5.031215283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:01.740017163 +0000 UTC m=+371.295645497" watchObservedRunningTime="2026-11-02 21:19:01.742006251 +0000 UTC m=+371.297634588" Nov 02 21:19:01.848721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:01.848669 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" podStartSLOduration=2.208389879 podStartE2EDuration="4.848654449s" totalImagesPullingTime="2.64026457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:01.847506597 +0000 UTC m=+371.403134933" watchObservedRunningTime="2026-11-02 21:19:01.848654449 +0000 UTC m=+371.404282785" Nov 02 21:19:02.225660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:02.225576 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:02.372257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:02.372224 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:02.500122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:02.500043 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:02.931798 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:02.931769 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb"] Nov 02 21:19:02.935051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:02.935031 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:02.988834 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:02.988808 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cbm2h\"" Nov 02 21:19:03.023763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.023737 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:03.038219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.038197 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb"] Nov 02 21:19:03.085648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.085615 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.085648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.085650 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.085831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.085723 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2469\" (UniqueName: \"kubernetes.io/projected/53fe7526-a649-4f45-8b6f-d1fb885ca908-kube-api-access-g2469\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.085831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.085760 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.085831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.085788 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.085941 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.085922 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.086005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.085977 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.086043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.086018 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe7526-a649-4f45-8b6f-d1fb885ca908-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.086075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.086039 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.161082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.161050 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:03.187256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187180 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187217 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187245 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2469\" (UniqueName: \"kubernetes.io/projected/53fe7526-a649-4f45-8b6f-d1fb885ca908-kube-api-access-g2469\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187276 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187305 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187364 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187392 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187433 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe7526-a649-4f45-8b6f-d1fb885ca908-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187472 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.187794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187593 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.188005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.187961 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.188086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.188045 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.188144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.188108 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.188278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.188257 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe7526-a649-4f45-8b6f-d1fb885ca908-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.189525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.189498 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.189833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.189814 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.241381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.241350 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fe7526-a649-4f45-8b6f-d1fb885ca908-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.241518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.241409 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2469\" (UniqueName: \"kubernetes.io/projected/53fe7526-a649-4f45-8b6f-d1fb885ca908-kube-api-access-g2469\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb\" (UID: \"53fe7526-a649-4f45-8b6f-d1fb885ca908\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.247583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.247167 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:03.290938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.290909 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:03.524168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.524086 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wz5w8" Nov 02 21:19:03.722104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.722071 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb"] Nov 02 21:19:03.728671 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:19:03.728644 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fe7526_a649_4f45_8b6f_d1fb885ca908.slice/crio-7e0302927b2e743435651fd2637781a24108daaed01cc9eb7580242564e61415 WatchSource:0}: Error finding container 7e0302927b2e743435651fd2637781a24108daaed01cc9eb7580242564e61415: Status 404 returned error can't find the container with id 7e0302927b2e743435651fd2637781a24108daaed01cc9eb7580242564e61415 Nov 02 21:19:03.730789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.730761 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:19:03.730864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.730823 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:19:03.730864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:03.730851 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:19:04.590065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:04.590034 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" event={"ID":"53fe7526-a649-4f45-8b6f-d1fb885ca908","Type":"ContainerStarted","Data":"70e8771662575be034cca1cd91c34ff002a7ac1053333684970e37926397dede"} Nov 02 21:19:04.590467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:04.590081 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" event={"ID":"53fe7526-a649-4f45-8b6f-d1fb885ca908","Type":"ContainerStarted","Data":"7e0302927b2e743435651fd2637781a24108daaed01cc9eb7580242564e61415"} Nov 02 21:19:04.753320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:04.753264 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" podStartSLOduration=2.753250641 podStartE2EDuration="2.753250641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:04.751599135 +0000 UTC m=+374.307227473" watchObservedRunningTime="2026-11-02 21:19:04.753250641 +0000 UTC m=+374.308878977" Nov 02 21:19:05.247847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:05.247816 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:05.252946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:05.252919 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:05.594983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:05.594949 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:05.595933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:05.595910 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb" Nov 02 21:19:06.311348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:06.311300 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:06.541699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:06.541665 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:06.661912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:06.661836 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:06.922529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:06.922457 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:07.040657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.040614 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:07.533336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.533300 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ndl5b"] Nov 02 21:19:07.539489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.539468 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" Nov 02 21:19:07.591910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.591884 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ndl5b"] Nov 02 21:19:07.627527 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.627505 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9wws\" (UniqueName: \"kubernetes.io/projected/364167f5-2765-468f-bdc1-cc2a77db2609-kube-api-access-h9wws\") pod \"authorino-57f7f8659f-ndl5b\" (UID: \"364167f5-2765-468f-bdc1-cc2a77db2609\") " pod="kuadrant-system/authorino-57f7f8659f-ndl5b" Nov 02 21:19:07.636469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.636452 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9s9fw\"" Nov 02 21:19:07.669628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.669600 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:07.728680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.728652 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9wws\" (UniqueName: \"kubernetes.io/projected/364167f5-2765-468f-bdc1-cc2a77db2609-kube-api-access-h9wws\") pod \"authorino-57f7f8659f-ndl5b\" (UID: \"364167f5-2765-468f-bdc1-cc2a77db2609\") " pod="kuadrant-system/authorino-57f7f8659f-ndl5b" Nov 02 21:19:07.782813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.782785 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9wws\" (UniqueName: \"kubernetes.io/projected/364167f5-2765-468f-bdc1-cc2a77db2609-kube-api-access-h9wws\") pod \"authorino-57f7f8659f-ndl5b\" (UID: \"364167f5-2765-468f-bdc1-cc2a77db2609\") " pod="kuadrant-system/authorino-57f7f8659f-ndl5b" Nov 02 21:19:07.801899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.801840 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:07.850154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:07.850133 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" Nov 02 21:19:08.213317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:08.213282 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ndl5b"] Nov 02 21:19:08.220216 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:19:08.220183 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod364167f5_2765_468f_bdc1_cc2a77db2609.slice/crio-815a15b0a71b046360f8db37fa11851add4b502d51033f3e1d1a00e85541dbf3 WatchSource:0}: Error finding container 815a15b0a71b046360f8db37fa11851add4b502d51033f3e1d1a00e85541dbf3: Status 404 returned error can't find the container with id 815a15b0a71b046360f8db37fa11851add4b502d51033f3e1d1a00e85541dbf3 Nov 02 21:19:08.610436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:08.610402 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" event={"ID":"364167f5-2765-468f-bdc1-cc2a77db2609","Type":"ContainerStarted","Data":"815a15b0a71b046360f8db37fa11851add4b502d51033f3e1d1a00e85541dbf3"} Nov 02 21:19:09.435288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.435250 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:09.611596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.611553 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:09.655012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.654986 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2"] Nov 02 21:19:09.658099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.658081 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.710181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.710037 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2"] Nov 02 21:19:09.710898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.710864 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 21:19:09.755135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.755109 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 21:19:09.755256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.755157 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bdwp9\"" Nov 02 21:19:09.849284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.849256 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e000b5aa-e884-4263-8bc7-1b64c302bd10-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.849402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.849298 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhvh\" (UniqueName: \"kubernetes.io/projected/e000b5aa-e884-4263-8bc7-1b64c302bd10-kube-api-access-fnhvh\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.849402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.849326 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e000b5aa-e884-4263-8bc7-1b64c302bd10-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.876509 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.876476 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:09.949931 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.949904 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e000b5aa-e884-4263-8bc7-1b64c302bd10-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.950068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.949944 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhvh\" (UniqueName: \"kubernetes.io/projected/e000b5aa-e884-4263-8bc7-1b64c302bd10-kube-api-access-fnhvh\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.950068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.949985 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e000b5aa-e884-4263-8bc7-1b64c302bd10-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.950569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.950536 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e000b5aa-e884-4263-8bc7-1b64c302bd10-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:09.952320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:09.952287 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e000b5aa-e884-4263-8bc7-1b64c302bd10-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:10.010809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:10.010756 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhvh\" (UniqueName: \"kubernetes.io/projected/e000b5aa-e884-4263-8bc7-1b64c302bd10-kube-api-access-fnhvh\") pod \"kuadrant-console-plugin-66c4fb575d-6d9v2\" (UID: \"e000b5aa-e884-4263-8bc7-1b64c302bd10\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:10.269470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:10.269374 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" Nov 02 21:19:10.616216 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:19:10.616185 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:19:10.647242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:10.647218 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2"] Nov 02 21:19:10.654626 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:19:10.654549 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode000b5aa_e884_4263_8bc7_1b64c302bd10.slice/crio-e9cabb1560f3b6f2dc51f3e85c3f07da53cc11047621ca085468e6576439df33 WatchSource:0}: Error finding container e9cabb1560f3b6f2dc51f3e85c3f07da53cc11047621ca085468e6576439df33: Status 404 returned error can't find the container with id e9cabb1560f3b6f2dc51f3e85c3f07da53cc11047621ca085468e6576439df33 Nov 02 21:19:11.632233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:11.632194 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" event={"ID":"e000b5aa-e884-4263-8bc7-1b64c302bd10","Type":"ContainerStarted","Data":"e9cabb1560f3b6f2dc51f3e85c3f07da53cc11047621ca085468e6576439df33"} Nov 02 21:19:12.583358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:12.583332 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4zzv5" Nov 02 21:19:12.583523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:12.583398 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 21:19:12.638607 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:12.638575 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" event={"ID":"364167f5-2765-468f-bdc1-cc2a77db2609","Type":"ContainerStarted","Data":"07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add"} Nov 02 21:19:13.619832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:13.619793 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:13.801432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:13.801338 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:14.864291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:14.864230 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" podStartSLOduration=4.277287616 podStartE2EDuration="7.864215253s" totalImagesPullingTime="3.586927637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:12.926030509 +0000 UTC m=+382.481658843" watchObservedRunningTime="2026-11-02 21:19:14.864215253 +0000 UTC m=+384.419843590" Nov 02 21:19:14.865548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:14.865525 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ndl5b"] Nov 02 21:19:14.865734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:14.865707 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" podUID="364167f5-2765-468f-bdc1-cc2a77db2609" containerName="authorino" containerID="cri-o://07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add" gracePeriod=30 Nov 02 21:19:15.124580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.124512 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" Nov 02 21:19:15.203550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.203517 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364167f5-2765-468f-bdc1-cc2a77db2609-kube-api-access-h9wws\" (UniqueName: \"kubernetes.io/projected/364167f5-2765-468f-bdc1-cc2a77db2609-kube-api-access-h9wws\") pod \"364167f5-2765-468f-bdc1-cc2a77db2609\" (UID: \"364167f5-2765-468f-bdc1-cc2a77db2609\") " Nov 02 21:19:15.206110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.206065 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364167f5-2765-468f-bdc1-cc2a77db2609-kube-api-access-h9wws" pod "364167f5-2765-468f-bdc1-cc2a77db2609" (UID: "364167f5-2765-468f-bdc1-cc2a77db2609"). InnerVolumeSpecName "kube-api-access-h9wws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:15.304904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.304867 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9wws\" (UniqueName: \"kubernetes.io/projected/364167f5-2765-468f-bdc1-cc2a77db2609-kube-api-access-h9wws\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:15.659263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.659229 4441 generic.go:356] "Generic (PLEG): container finished" podID="364167f5-2765-468f-bdc1-cc2a77db2609" containerID="07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add" exitCode=0 Nov 02 21:19:15.659443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.659274 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" event={"ID":"364167f5-2765-468f-bdc1-cc2a77db2609","Type":"ContainerDied","Data":"07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add"} Nov 02 21:19:15.659443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.659289 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" Nov 02 21:19:15.659443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.659317 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ndl5b" event={"ID":"364167f5-2765-468f-bdc1-cc2a77db2609","Type":"ContainerDied","Data":"815a15b0a71b046360f8db37fa11851add4b502d51033f3e1d1a00e85541dbf3"} Nov 02 21:19:15.659443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.659338 4441 scope.go:122] "RemoveContainer" containerID="07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add" Nov 02 21:19:15.669940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.669917 4441 scope.go:122] "RemoveContainer" containerID="07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add" Nov 02 21:19:15.670241 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:19:15.670215 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add\": container with ID starting with 07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add not found: ID does not exist" containerID="07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add" Nov 02 21:19:15.670328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.670257 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add"} err="failed to get container status \"07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add\": rpc error: code = NotFound desc = could not find container \"07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add\": container with ID starting with 07fc96367d84761b65cb8ca825f7b662e5877ca432b9b5cf994682c1bace2add not found: ID does not exist" Nov 02 21:19:15.822632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.822592 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ndl5b"] Nov 02 21:19:15.860360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:15.860328 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ndl5b"] Nov 02 21:19:16.380597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:16.380548 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:16.500809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:16.500770 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:16.583739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:16.583706 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364167f5-2765-468f-bdc1-cc2a77db2609" path="/var/lib/kubelet/pods/364167f5-2765-468f-bdc1-cc2a77db2609/volumes" Nov 02 21:19:16.617461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:16.617437 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:17.672722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.672682 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" event={"ID":"e000b5aa-e884-4263-8bc7-1b64c302bd10","Type":"ContainerStarted","Data":"ad81127a36cc4ca2522bfe80b3f51e1e7e5c17807e5235c3f3f737d9fcd8531c"} Nov 02 21:19:17.785008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.784939 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6d9v2" podStartSLOduration=2.7868640940000002 podStartE2EDuration="8.784926471s" totalImagesPullingTime="5.998062377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:17.783580424 +0000 UTC m=+387.339208773" watchObservedRunningTime="2026-11-02 21:19:17.784926471 +0000 UTC m=+387.340554808" Nov 02 21:19:17.938820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.938744 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7"] Nov 02 21:19:17.939109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.939097 4441 cpu_manager.go:414] "containerMap: removing container" podUID="364167f5-2765-468f-bdc1-cc2a77db2609" containerName="authorino" Nov 02 21:19:17.939155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.939111 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364167f5-2765-468f-bdc1-cc2a77db2609" containerName="authorino" Nov 02 21:19:17.939189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.939158 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="364167f5-2765-468f-bdc1-cc2a77db2609" containerName="authorino" Nov 02 21:19:17.942383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.942364 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:17.992025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.992006 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 21:19:17.992691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.992668 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 21:19:17.992796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.992737 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:19:17.992859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.992843 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 21:19:17.995812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:17.995792 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7"] Nov 02 21:19:18.026577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.026550 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-metrics-certs\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.026676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.026583 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-manager-config\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.026676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.026616 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqbk\" (UniqueName: \"kubernetes.io/projected/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-kube-api-access-gmqbk\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.026785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.026764 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-cert\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.039484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.039464 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-hb626\"" Nov 02 21:19:18.039550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.039506 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 21:19:18.127683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.127652 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-cert\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.127825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.127695 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-metrics-certs\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.127825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.127716 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-manager-config\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.127825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.127750 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqbk\" (UniqueName: \"kubernetes.io/projected/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-kube-api-access-gmqbk\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.128469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.128448 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-manager-config\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.130134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.130112 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-cert\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.130202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.130164 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-metrics-certs\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.181473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.181444 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqbk\" (UniqueName: \"kubernetes.io/projected/340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba-kube-api-access-gmqbk\") pod \"jobset-controller-manager-6d695b7c7f-jrcv7\" (UID: \"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba\") " pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.254377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.254313 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:18.621265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.621225 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7"] Nov 02 21:19:18.628074 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:19:18.628043 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340b7ccd_6f3f_4c19_bea0_d9c5cf9bc4ba.slice/crio-3f3726bb1100767c3f9bbdeeede5aaf4f6a3b5d42cbc08a614394795e54b359c WatchSource:0}: Error finding container 3f3726bb1100767c3f9bbdeeede5aaf4f6a3b5d42cbc08a614394795e54b359c: Status 404 returned error can't find the container with id 3f3726bb1100767c3f9bbdeeede5aaf4f6a3b5d42cbc08a614394795e54b359c Nov 02 21:19:18.679035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.679002 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" event={"ID":"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba","Type":"ContainerStarted","Data":"3f3726bb1100767c3f9bbdeeede5aaf4f6a3b5d42cbc08a614394795e54b359c"} Nov 02 21:19:18.954890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:18.954807 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:19.075653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:19.075618 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:19.558274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:19.558243 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:19.681737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:19.681696 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:22.956248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:22.956211 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:23.071386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:23.071358 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:23.373792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:23.373758 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:23.501066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:23.501037 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:23.707080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:23.707001 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" event={"ID":"340b7ccd-6f3f-4c19-bea0-d9c5cf9bc4ba","Type":"ContainerStarted","Data":"f02a257ee3db574bd1279a49e7e07110750e177cd7fe4e82a20a33eaf3eb8c2a"} Nov 02 21:19:23.707209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:23.707080 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:23.815835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:23.815782 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" podStartSLOduration=2.472441754 podStartE2EDuration="6.815767185s" totalImagesPullingTime="4.343325431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:23.813829569 +0000 UTC m=+393.369457916" watchObservedRunningTime="2026-11-02 21:19:23.815767185 +0000 UTC m=+393.371395522" Nov 02 21:19:27.212081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:27.212039 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:27.364372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:27.364317 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:27.512624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:27.512531 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:27.750331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:27.750293 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:27.880113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:27.880084 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:28.011014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:28.010960 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:33.422304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:33.422230 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:33.540320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:33.540287 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:34.716670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:34.716646 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6d695b7c7f-jrcv7" Nov 02 21:19:40.613430 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:19:40.613393 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:19:42.796886 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:42.796851 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:49.373152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:49.373108 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:49.501912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:49.501877 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:49.632014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:49.631912 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:49.769656 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:49.769624 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:53.071317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:53.071278 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:53.187566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:53.187531 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:54.381019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:54.380960 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:54.503910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:54.503878 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:57.326358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:57.326323 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:57.486029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:19:57.485992 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:10.613547 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:20:10.613512 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:20:19.819104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:19.819069 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:19.948811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:19.948763 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.073251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.073150 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.195121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.195085 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.321958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.321930 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.450199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.450125 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.573696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.573651 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.700657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.700563 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.819814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.819781 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:20.950381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:20.950345 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:21.944451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:21.944404 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:22.109819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:22.109788 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:25.749493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:25.749461 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:25.873026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:25.872990 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:27.424090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:27.424059 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:27.545670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:27.545634 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:27.672723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:27.672697 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:27.796544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:27.796467 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:27.930514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:27.930477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:28.057472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:28.057428 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:28.177397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:28.177361 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:28.318876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:28.318799 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:28.440555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:28.440519 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:28.558040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:28.558008 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:28.685990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:28.685893 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:33.913143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:33.913109 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:34.033760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:34.033726 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:39.616306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:39.616270 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:39.820855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:39.820822 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:40.274316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.274282 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fc498fc56-2fj2f"] Nov 02 21:20:40.277155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.277135 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" Nov 02 21:20:40.330025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.329994 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fc498fc56-2fj2f"] Nov 02 21:20:40.373759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.373736 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9s9fw\"" Nov 02 21:20:40.398349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.398323 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vg7\" (UniqueName: \"kubernetes.io/projected/e7d9dab9-2db0-4f48-aa80-7d3b81db020e-kube-api-access-w6vg7\") pod \"authorino-7fc498fc56-2fj2f\" (UID: \"e7d9dab9-2db0-4f48-aa80-7d3b81db020e\") " pod="kuadrant-system/authorino-7fc498fc56-2fj2f" Nov 02 21:20:40.498892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.498863 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vg7\" (UniqueName: \"kubernetes.io/projected/e7d9dab9-2db0-4f48-aa80-7d3b81db020e-kube-api-access-w6vg7\") pod \"authorino-7fc498fc56-2fj2f\" (UID: \"e7d9dab9-2db0-4f48-aa80-7d3b81db020e\") " pod="kuadrant-system/authorino-7fc498fc56-2fj2f" Nov 02 21:20:40.553593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.553519 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vg7\" (UniqueName: \"kubernetes.io/projected/e7d9dab9-2db0-4f48-aa80-7d3b81db020e-kube-api-access-w6vg7\") pod \"authorino-7fc498fc56-2fj2f\" (UID: \"e7d9dab9-2db0-4f48-aa80-7d3b81db020e\") " pod="kuadrant-system/authorino-7fc498fc56-2fj2f" Nov 02 21:20:40.588152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.588126 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" Nov 02 21:20:40.621705 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:20:40.621650 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:20:40.724722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.724688 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fc498fc56-2fj2f"] Nov 02 21:20:40.751181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.751154 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c8545c7db-pfbg9"] Nov 02 21:20:40.754994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.754955 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:40.801428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.801389 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-tls-cert\") pod \"authorino-5c8545c7db-pfbg9\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:40.801630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.801447 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwv2z\" (UniqueName: \"kubernetes.io/projected/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-kube-api-access-vwv2z\") pod \"authorino-5c8545c7db-pfbg9\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:40.804796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.804726 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 21:20:40.808284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.808217 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c8545c7db-pfbg9"] Nov 02 21:20:40.902087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.902047 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-tls-cert\") pod \"authorino-5c8545c7db-pfbg9\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:40.902241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.902105 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwv2z\" (UniqueName: \"kubernetes.io/projected/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-kube-api-access-vwv2z\") pod \"authorino-5c8545c7db-pfbg9\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:40.904515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.904480 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-tls-cert\") pod \"authorino-5c8545c7db-pfbg9\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:40.955727 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.955695 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwv2z\" (UniqueName: \"kubernetes.io/projected/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-kube-api-access-vwv2z\") pod \"authorino-5c8545c7db-pfbg9\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:40.963656 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:40.963633 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fc498fc56-2fj2f"] Nov 02 21:20:40.970521 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:20:40.970499 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7d9dab9_2db0_4f48_aa80_7d3b81db020e.slice/crio-8958b24f50671800bd87170e41e1f172682a41ea10554eab9bf4b8ad5a635e95 WatchSource:0}: Error finding container 8958b24f50671800bd87170e41e1f172682a41ea10554eab9bf4b8ad5a635e95: Status 404 returned error can't find the container with id 8958b24f50671800bd87170e41e1f172682a41ea10554eab9bf4b8ad5a635e95 Nov 02 21:20:41.068175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:41.068147 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:20:41.110590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:41.110552 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" event={"ID":"e7d9dab9-2db0-4f48-aa80-7d3b81db020e","Type":"ContainerStarted","Data":"8958b24f50671800bd87170e41e1f172682a41ea10554eab9bf4b8ad5a635e95"} Nov 02 21:20:41.436336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:41.436256 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c8545c7db-pfbg9"] Nov 02 21:20:41.444035 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:20:41.444006 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fac2aaf_5b4c_4321_b329_022bc6ea64d8.slice/crio-3fc9268973a41e9001bf7468449ab6871151abc5801b2862cdc8e98415ae1e5b WatchSource:0}: Error finding container 3fc9268973a41e9001bf7468449ab6871151abc5801b2862cdc8e98415ae1e5b: Status 404 returned error can't find the container with id 3fc9268973a41e9001bf7468449ab6871151abc5801b2862cdc8e98415ae1e5b Nov 02 21:20:42.117906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:42.117862 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" event={"ID":"7fac2aaf-5b4c-4321-b329-022bc6ea64d8","Type":"ContainerStarted","Data":"3fc9268973a41e9001bf7468449ab6871151abc5801b2862cdc8e98415ae1e5b"} Nov 02 21:20:43.122043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.121917 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:43.124863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.124829 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" event={"ID":"e7d9dab9-2db0-4f48-aa80-7d3b81db020e","Type":"ContainerStarted","Data":"3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d"} Nov 02 21:20:43.125215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.124903 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" podUID="e7d9dab9-2db0-4f48-aa80-7d3b81db020e" containerName="authorino" containerID="cri-o://3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d" gracePeriod=30 Nov 02 21:20:43.126760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.126734 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" event={"ID":"7fac2aaf-5b4c-4321-b329-022bc6ea64d8","Type":"ContainerStarted","Data":"c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044"} Nov 02 21:20:43.234521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.234462 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" podStartSLOduration=1.709472806 podStartE2EDuration="3.234449856s" totalImagesPullingTime="1.52497705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:43.23258772 +0000 UTC m=+472.788216058" watchObservedRunningTime="2026-11-02 21:20:43.234449856 +0000 UTC m=+472.790078193" Nov 02 21:20:43.340777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.340729 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" podStartSLOduration=1.9660931910000001 podStartE2EDuration="3.340716507s" totalImagesPullingTime="1.374623316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:43.338559875 +0000 UTC m=+472.894188211" watchObservedRunningTime="2026-11-02 21:20:43.340716507 +0000 UTC m=+472.896344844" Nov 02 21:20:43.380655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.373256 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" Nov 02 21:20:43.423227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.423192 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d9dab9-2db0-4f48-aa80-7d3b81db020e-kube-api-access-w6vg7\" (UniqueName: \"kubernetes.io/projected/e7d9dab9-2db0-4f48-aa80-7d3b81db020e-kube-api-access-w6vg7\") pod \"e7d9dab9-2db0-4f48-aa80-7d3b81db020e\" (UID: \"e7d9dab9-2db0-4f48-aa80-7d3b81db020e\") " Nov 02 21:20:43.425493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.425467 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d9dab9-2db0-4f48-aa80-7d3b81db020e-kube-api-access-w6vg7" pod "e7d9dab9-2db0-4f48-aa80-7d3b81db020e" (UID: "e7d9dab9-2db0-4f48-aa80-7d3b81db020e"). InnerVolumeSpecName "kube-api-access-w6vg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:20:43.523855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:43.523823 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vg7\" (UniqueName: \"kubernetes.io/projected/e7d9dab9-2db0-4f48-aa80-7d3b81db020e-kube-api-access-w6vg7\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:44.133958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.133886 4441 generic.go:356] "Generic (PLEG): container finished" podID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerID="47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b" exitCode=0 Nov 02 21:20:44.134329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.133962 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" event={"ID":"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190","Type":"ContainerDied","Data":"47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b"} Nov 02 21:20:44.134417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.134391 4441 scope.go:122] "RemoveContainer" containerID="47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b" Nov 02 21:20:44.135347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135326 4441 generic.go:356] "Generic (PLEG): container finished" podID="e7d9dab9-2db0-4f48-aa80-7d3b81db020e" containerID="3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d" exitCode=0 Nov 02 21:20:44.135457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135400 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" event={"ID":"e7d9dab9-2db0-4f48-aa80-7d3b81db020e","Type":"ContainerDied","Data":"3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d"} Nov 02 21:20:44.135457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135445 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" event={"ID":"e7d9dab9-2db0-4f48-aa80-7d3b81db020e","Type":"ContainerDied","Data":"8958b24f50671800bd87170e41e1f172682a41ea10554eab9bf4b8ad5a635e95"} Nov 02 21:20:44.135576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135466 4441 scope.go:122] "RemoveContainer" containerID="3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d" Nov 02 21:20:44.135576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135409 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fc498fc56-2fj2f" Nov 02 21:20:44.135576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135526 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:20:44.135722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135603 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:20:44.135722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.135651 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:20:44.148691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.148672 4441 scope.go:122] "RemoveContainer" containerID="3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d" Nov 02 21:20:44.148963 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:20:44.148947 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d\": container with ID starting with 3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d not found: ID does not exist" containerID="3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d" Nov 02 21:20:44.149036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.148983 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d"} err="failed to get container status \"3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d\": rpc error: code = NotFound desc = could not find container \"3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d\": container with ID starting with 3d9145dc43d3ef3770e3099c42c1625333520f9f9822e9c36cb9bdcbe4e83d2d not found: ID does not exist" Nov 02 21:20:44.357438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.357401 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fc498fc56-2fj2f"] Nov 02 21:20:44.359429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.359407 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7fc498fc56-2fj2f"] Nov 02 21:20:44.581659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:44.581628 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d9dab9-2db0-4f48-aa80-7d3b81db020e" path="/var/lib/kubelet/pods/e7d9dab9-2db0-4f48-aa80-7d3b81db020e/volumes" Nov 02 21:20:45.143041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:45.143004 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" event={"ID":"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190","Type":"ContainerStarted","Data":"c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228"} Nov 02 21:20:45.656079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:45.656046 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:20:45.656283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:45.656216 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 02 21:20:45.656356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:45.656272 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 02 21:20:46.655953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:46.655919 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 02 21:20:46.656336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:46.655992 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 02 21:20:47.656098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:47.656065 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 02 21:20:47.656543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:47.656120 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 02 21:20:48.656061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:48.656034 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 02 21:20:48.656278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:48.656085 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 02 21:20:49.655982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:49.655942 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 02 21:20:49.656150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:49.656009 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 02 21:20:50.659469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:50.659443 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:20:50.659803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:50.659691 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:20:50.660400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:20:50.660380 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:21:10.613170 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:21:10.613084 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:21:13.605618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:21:13.605585 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:13.741562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:21:13.741533 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:21.965688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:21:21.965652 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:40.613226 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:21:40.613187 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:21:43.457141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:21:43.457110 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:10.613181 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:22:10.613149 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:22:18.887093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:18.887057 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:19.021144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:19.021113 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:36.892407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:36.892372 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:40.626023 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:22:40.625984 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:22:41.695834 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:41.695797 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:43.807143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:43.807107 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:45.746678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:45.746645 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:49.319987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:49.319918 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:50.611888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:50.611864 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:22:50.614705 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:50.614672 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:22:50.628109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:50.628089 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:22:50.631008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:50.630958 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:22:50.638902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:50.638884 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:22:50.641959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:50.641941 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:22:53.316800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:53.316760 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:56.740802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.740768 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh"] Nov 02 21:22:56.741215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.741107 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e7d9dab9-2db0-4f48-aa80-7d3b81db020e" containerName="authorino" Nov 02 21:22:56.741215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.741118 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d9dab9-2db0-4f48-aa80-7d3b81db020e" containerName="authorino" Nov 02 21:22:56.741215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.741170 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d9dab9-2db0-4f48-aa80-7d3b81db020e" containerName="authorino" Nov 02 21:22:56.744227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.744209 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.797301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.797271 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh"] Nov 02 21:22:56.823812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.823784 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:56.841796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.841767 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 21:22:56.841796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.841789 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 21:22:56.842022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.841794 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 21:22:56.842022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.841834 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8m4js\"" Nov 02 21:22:56.892163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.892125 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpv4q\" (UniqueName: \"kubernetes.io/projected/1ca7a877-0358-45c1-949e-b32728684968-kube-api-access-lpv4q\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.892328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.892201 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.892328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.892240 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.892328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.892314 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.892491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.892351 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca7a877-0358-45c1-949e-b32728684968-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.892491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.892385 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.993562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.993478 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.993562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.993519 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.993562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.993537 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca7a877-0358-45c1-949e-b32728684968-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.993782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.993574 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.993782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.993637 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpv4q\" (UniqueName: \"kubernetes.io/projected/1ca7a877-0358-45c1-949e-b32728684968-kube-api-access-lpv4q\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.993782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.993688 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.994052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.994029 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.994112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.994071 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.994112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.994086 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.995864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.995842 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca7a877-0358-45c1-949e-b32728684968-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:56.996075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:56.996058 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca7a877-0358-45c1-949e-b32728684968-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:57.047397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:57.047373 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpv4q\" (UniqueName: \"kubernetes.io/projected/1ca7a877-0358-45c1-949e-b32728684968-kube-api-access-lpv4q\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh\" (UID: \"1ca7a877-0358-45c1-949e-b32728684968\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:57.055227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:57.055205 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:22:57.420241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:57.420202 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh"] Nov 02 21:22:57.426642 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:22:57.426617 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ca7a877_0358_45c1_949e_b32728684968.slice/crio-8a3b6265bbb66e71ecfed672362979ca99ae14ec93483ca84efa68e9002984bf WatchSource:0}: Error finding container 8a3b6265bbb66e71ecfed672362979ca99ae14ec93483ca84efa68e9002984bf: Status 404 returned error can't find the container with id 8a3b6265bbb66e71ecfed672362979ca99ae14ec93483ca84efa68e9002984bf Nov 02 21:22:57.427904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:57.427888 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:22:57.798694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:57.798617 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" event={"ID":"1ca7a877-0358-45c1-949e-b32728684968","Type":"ContainerStarted","Data":"8a3b6265bbb66e71ecfed672362979ca99ae14ec93483ca84efa68e9002984bf"} Nov 02 21:22:59.799382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.799351 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69"] Nov 02 21:22:59.802881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.802858 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:22:59.811846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.811821 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" event={"ID":"1ca7a877-0358-45c1-949e-b32728684968","Type":"ContainerStarted","Data":"731a491183da023cb63374e1d24895f1ae0f80d98ab6d36f146b78cbb0f62f00"} Nov 02 21:22:59.892847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.892816 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:59.901120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.901094 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 21:22:59.903464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.903440 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69"] Nov 02 21:22:59.919913 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.919890 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:22:59.920025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.919946 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:22:59.920025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.919983 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f6edef-f667-44fa-8cea-f87ba914b7b8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:22:59.920195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.920178 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:22:59.920238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.920207 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2dc\" (UniqueName: \"kubernetes.io/projected/30f6edef-f667-44fa-8cea-f87ba914b7b8-kube-api-access-nz2dc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:22:59.920278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:22:59.920246 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.020891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.020855 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.020891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.020897 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f6edef-f667-44fa-8cea-f87ba914b7b8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.021215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.020963 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.021215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.021020 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2dc\" (UniqueName: \"kubernetes.io/projected/30f6edef-f667-44fa-8cea-f87ba914b7b8-kube-api-access-nz2dc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.021215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.021075 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.021215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.021108 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.022066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.022022 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.022211 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.022176 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.022290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.022272 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.023217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.023174 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podStartSLOduration=2.502552442 podStartE2EDuration="4.023161001s" totalImagesPullingTime="1.520608559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:00.021026422 +0000 UTC m=+609.576654776" watchObservedRunningTime="2026-11-02 21:23:00.023161001 +0000 UTC m=+609.578789362" Nov 02 21:23:00.024273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.024203 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30f6edef-f667-44fa-8cea-f87ba914b7b8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.027341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.027324 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f6edef-f667-44fa-8cea-f87ba914b7b8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.055920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.055868 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:23:00.077934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.077903 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2dc\" (UniqueName: \"kubernetes.io/projected/30f6edef-f667-44fa-8cea-f87ba914b7b8-kube-api-access-nz2dc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tng69\" (UID: \"30f6edef-f667-44fa-8cea-f87ba914b7b8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.115881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.115860 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:00.483772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.483734 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69"] Nov 02 21:23:00.489381 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:23:00.489352 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f6edef_f667_44fa_8cea_f87ba914b7b8.slice/crio-0a9110c45a72985c3a2d8628c75755633ad033bcb2d02d88f449e741abb1c75f WatchSource:0}: Error finding container 0a9110c45a72985c3a2d8628c75755633ad033bcb2d02d88f449e741abb1c75f: Status 404 returned error can't find the container with id 0a9110c45a72985c3a2d8628c75755633ad033bcb2d02d88f449e741abb1c75f Nov 02 21:23:00.821193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:00.821152 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" event={"ID":"30f6edef-f667-44fa-8cea-f87ba914b7b8","Type":"ContainerStarted","Data":"0a9110c45a72985c3a2d8628c75755633ad033bcb2d02d88f449e741abb1c75f"} Nov 02 21:23:01.827795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:01.827758 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" event={"ID":"30f6edef-f667-44fa-8cea-f87ba914b7b8","Type":"ContainerStarted","Data":"662aad6d15c4860c59df75c1bf48644a2ed0c74b0191a53ddb9c9bc7f19abaa6"} Nov 02 21:23:01.938086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:01.938027 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" podStartSLOduration=2.346761685 podStartE2EDuration="2.938010373s" totalImagesPullingTime="591.248688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:01.935795157 +0000 UTC m=+611.491423496" watchObservedRunningTime="2026-11-02 21:23:01.938010373 +0000 UTC m=+611.493638710" Nov 02 21:23:02.117114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:02.116999 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:07.055595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.055555 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:23:07.068077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.068051 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:23:07.160510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.160467 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:07.858689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.858657 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2"] Nov 02 21:23:07.862125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.862099 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.870293 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.870269 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:23:07.992828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.992800 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.992993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.992833 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.992993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.992862 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.993088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.993061 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.993157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.993089 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.993213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.993153 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.993266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.993249 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.993638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.993278 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9qt\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-kube-api-access-7g9qt\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:07.993638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:07.993313 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.051959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.051925 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2"] Nov 02 21:23:08.094400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094362 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.094400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094409 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9qt\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-kube-api-access-7g9qt\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094462 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094500 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094528 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094572 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094608 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094646 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094672 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.094800 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.095090 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.095173 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.095187 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.095454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.095435 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.097853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.097830 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.098119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.098097 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.148452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.148389 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9qt\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-kube-api-access-7g9qt\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.148822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.148804 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rp6l2\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.174737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.174711 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:08.596724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.596690 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2"] Nov 02 21:23:08.603432 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:23:08.603408 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf0e9e9_98cd_4e04_a71a_e028bdb73713.slice/crio-d66fb3b708e0ccf74ca6f32848c1bc435bd234331c528dcbbbe1c258dc2f1b12 WatchSource:0}: Error finding container d66fb3b708e0ccf74ca6f32848c1bc435bd234331c528dcbbbe1c258dc2f1b12: Status 404 returned error can't find the container with id d66fb3b708e0ccf74ca6f32848c1bc435bd234331c528dcbbbe1c258dc2f1b12 Nov 02 21:23:08.605361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.605335 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:23:08.605434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.605395 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:23:08.605434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.605425 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:23:08.867488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.867390 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" event={"ID":"fdf0e9e9-98cd-4e04-a71a-e028bdb73713","Type":"ContainerStarted","Data":"a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495"} Nov 02 21:23:08.867488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.867449 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" event={"ID":"fdf0e9e9-98cd-4e04-a71a-e028bdb73713","Type":"ContainerStarted","Data":"d66fb3b708e0ccf74ca6f32848c1bc435bd234331c528dcbbbe1c258dc2f1b12"} Nov 02 21:23:08.977883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:08.977834 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podStartSLOduration=1.977820339 podStartE2EDuration="1.977820339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:08.976835902 +0000 UTC m=+618.532464240" watchObservedRunningTime="2026-11-02 21:23:08.977820339 +0000 UTC m=+618.533448675" Nov 02 21:23:09.191053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:09.182333 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:09.191053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:09.189081 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:23:09.191053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:09.189153 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 21:23:10.116312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:10.116260 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:10.128510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:10.128478 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:10.177992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:10.175269 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:23:10.177992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:10.175333 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 21:23:10.625495 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:23:10.625459 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:23:10.912401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:10.912321 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" Nov 02 21:23:11.176858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:11.176326 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:23:11.176858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:11.176406 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 21:23:12.178345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:12.178306 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:23:12.179783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:12.179746 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 21:23:13.182352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:13.182314 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:23:13.182904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:13.182874 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 21:23:14.189990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:14.189353 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:23:14.189990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:14.189432 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 21:23:15.175129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:15.175088 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:23:15.175312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:15.175157 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 21:23:17.217148 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:17.217108 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:23:17.217652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:17.217180 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:23:18.259122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:18.259089 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:23:18.259593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:18.259139 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:23:18.326062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:18.326029 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:18.326447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:18.326428 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:18.327053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:18.327032 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:23:20.490359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:20.490299 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:26.937313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:26.937278 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74c57f8d5-mq6lj"] Nov 02 21:23:26.941263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:26.941241 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:26.956696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:26.956670 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshhf\" (UniqueName: \"kubernetes.io/projected/33243403-0984-43ce-90f3-04befaf56f63-kube-api-access-lshhf\") pod \"authorino-74c57f8d5-mq6lj\" (UID: \"33243403-0984-43ce-90f3-04befaf56f63\") " pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:26.956815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:26.956730 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/33243403-0984-43ce-90f3-04befaf56f63-tls-cert\") pod \"authorino-74c57f8d5-mq6lj\" (UID: \"33243403-0984-43ce-90f3-04befaf56f63\") " pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:26.992595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:26.992571 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74c57f8d5-mq6lj"] Nov 02 21:23:27.057620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.057586 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshhf\" (UniqueName: \"kubernetes.io/projected/33243403-0984-43ce-90f3-04befaf56f63-kube-api-access-lshhf\") pod \"authorino-74c57f8d5-mq6lj\" (UID: \"33243403-0984-43ce-90f3-04befaf56f63\") " pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:27.057770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.057692 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/33243403-0984-43ce-90f3-04befaf56f63-tls-cert\") pod \"authorino-74c57f8d5-mq6lj\" (UID: \"33243403-0984-43ce-90f3-04befaf56f63\") " pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:27.060613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.060583 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/33243403-0984-43ce-90f3-04befaf56f63-tls-cert\") pod \"authorino-74c57f8d5-mq6lj\" (UID: \"33243403-0984-43ce-90f3-04befaf56f63\") " pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:27.119025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.118998 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshhf\" (UniqueName: \"kubernetes.io/projected/33243403-0984-43ce-90f3-04befaf56f63-kube-api-access-lshhf\") pod \"authorino-74c57f8d5-mq6lj\" (UID: \"33243403-0984-43ce-90f3-04befaf56f63\") " pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:27.142302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.142243 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:27.253205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.253136 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74c57f8d5-mq6lj" Nov 02 21:23:27.416612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.416580 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:27.620481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:27.620444 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74c57f8d5-mq6lj"] Nov 02 21:23:27.627087 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:23:27.627061 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33243403_0984_43ce_90f3_04befaf56f63.slice/crio-1d41a489b44ee9461d293af67f098b4193b230a526ee600ba874419d7e7fb6f9 WatchSource:0}: Error finding container 1d41a489b44ee9461d293af67f098b4193b230a526ee600ba874419d7e7fb6f9: Status 404 returned error can't find the container with id 1d41a489b44ee9461d293af67f098b4193b230a526ee600ba874419d7e7fb6f9 Nov 02 21:23:28.001945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:28.001865 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74c57f8d5-mq6lj" event={"ID":"33243403-0984-43ce-90f3-04befaf56f63","Type":"ContainerStarted","Data":"1d41a489b44ee9461d293af67f098b4193b230a526ee600ba874419d7e7fb6f9"} Nov 02 21:23:29.009532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.009227 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74c57f8d5-mq6lj" event={"ID":"33243403-0984-43ce-90f3-04befaf56f63","Type":"ContainerStarted","Data":"e92f90b27740752c21b657386482f108f0185266a318b18ff65cb266d6f4bd82"} Nov 02 21:23:29.115699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.115648 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74c57f8d5-mq6lj" podStartSLOduration=1.816621779 podStartE2EDuration="3.11563212s" totalImagesPullingTime="1.299010341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:29.115144158 +0000 UTC m=+638.670772497" watchObservedRunningTime="2026-11-02 21:23:29.11563212 +0000 UTC m=+638.671260484" Nov 02 21:23:29.141056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.141017 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c8545c7db-pfbg9"] Nov 02 21:23:29.141291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.141241 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" podUID="7fac2aaf-5b4c-4321-b329-022bc6ea64d8" containerName="authorino" containerID="cri-o://c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044" gracePeriod=30 Nov 02 21:23:29.391157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.391134 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:23:29.479719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.479640 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-tls-cert\" (UniqueName: \"kubernetes.io/secret/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-tls-cert\") pod \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " Nov 02 21:23:29.479719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.479701 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-kube-api-access-vwv2z\" (UniqueName: \"kubernetes.io/projected/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-kube-api-access-vwv2z\") pod \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\" (UID: \"7fac2aaf-5b4c-4321-b329-022bc6ea64d8\") " Nov 02 21:23:29.482114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.482040 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-kube-api-access-vwv2z" pod "7fac2aaf-5b4c-4321-b329-022bc6ea64d8" (UID: "7fac2aaf-5b4c-4321-b329-022bc6ea64d8"). InnerVolumeSpecName "kube-api-access-vwv2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:23:29.493401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.493364 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-tls-cert" pod "7fac2aaf-5b4c-4321-b329-022bc6ea64d8" (UID: "7fac2aaf-5b4c-4321-b329-022bc6ea64d8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:23:29.581403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.581366 4441 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-tls-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:29.581403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:29.581396 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwv2z\" (UniqueName: \"kubernetes.io/projected/7fac2aaf-5b4c-4321-b329-022bc6ea64d8-kube-api-access-vwv2z\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:30.016651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.016618 4441 generic.go:356] "Generic (PLEG): container finished" podID="7fac2aaf-5b4c-4321-b329-022bc6ea64d8" containerID="c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044" exitCode=0 Nov 02 21:23:30.017147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.016668 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" Nov 02 21:23:30.017147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.016706 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" event={"ID":"7fac2aaf-5b4c-4321-b329-022bc6ea64d8","Type":"ContainerDied","Data":"c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044"} Nov 02 21:23:30.017147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.016750 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c8545c7db-pfbg9" event={"ID":"7fac2aaf-5b4c-4321-b329-022bc6ea64d8","Type":"ContainerDied","Data":"3fc9268973a41e9001bf7468449ab6871151abc5801b2862cdc8e98415ae1e5b"} Nov 02 21:23:30.017147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.016770 4441 scope.go:122] "RemoveContainer" containerID="c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044" Nov 02 21:23:30.027574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.027552 4441 scope.go:122] "RemoveContainer" containerID="c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044" Nov 02 21:23:30.027905 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:23:30.027879 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044\": container with ID starting with c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044 not found: ID does not exist" containerID="c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044" Nov 02 21:23:30.028021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.027915 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044"} err="failed to get container status \"c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044\": rpc error: code = NotFound desc = could not find container \"c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044\": container with ID starting with c86430e74ed855911d4df23df3ee52e61ac443b773d77306ea8a82aef6a30044 not found: ID does not exist" Nov 02 21:23:30.172415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.172383 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c8545c7db-pfbg9"] Nov 02 21:23:30.175605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.175582 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c8545c7db-pfbg9"] Nov 02 21:23:30.594040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:30.594001 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fac2aaf-5b4c-4321-b329-022bc6ea64d8" path="/var/lib/kubelet/pods/7fac2aaf-5b4c-4321-b329-022bc6ea64d8/volumes" Nov 02 21:23:34.167192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:34.167147 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:34.336266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:34.336211 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:40.613953 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:23:40.613919 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:23:42.281840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.281810 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6"] Nov 02 21:23:42.282259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.282185 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7fac2aaf-5b4c-4321-b329-022bc6ea64d8" containerName="authorino" Nov 02 21:23:42.282259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.282198 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fac2aaf-5b4c-4321-b329-022bc6ea64d8" containerName="authorino" Nov 02 21:23:42.282338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.282267 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fac2aaf-5b4c-4321-b329-022bc6ea64d8" containerName="authorino" Nov 02 21:23:42.285140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.285117 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.382364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.382337 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 21:23:42.382364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.382343 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 21:23:42.382588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.382391 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 21:23:42.382588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.382471 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-2vc2g\"" Nov 02 21:23:42.382588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.382517 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 21:23:42.386399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.386375 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6"] Nov 02 21:23:42.411049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.411021 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccskn\" (UniqueName: \"kubernetes.io/projected/636400c6-9f4c-476d-964b-2bf5a279c586-kube-api-access-ccskn\") pod \"ai-gateway-controller-7bc68ffc7c-pmmp6\" (UID: \"636400c6-9f4c-476d-964b-2bf5a279c586\") " pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.411206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.411186 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636400c6-9f4c-476d-964b-2bf5a279c586-metrics-certs\") pod \"ai-gateway-controller-7bc68ffc7c-pmmp6\" (UID: \"636400c6-9f4c-476d-964b-2bf5a279c586\") " pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.511729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.511693 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636400c6-9f4c-476d-964b-2bf5a279c586-metrics-certs\") pod \"ai-gateway-controller-7bc68ffc7c-pmmp6\" (UID: \"636400c6-9f4c-476d-964b-2bf5a279c586\") " pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.511832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.511740 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccskn\" (UniqueName: \"kubernetes.io/projected/636400c6-9f4c-476d-964b-2bf5a279c586-kube-api-access-ccskn\") pod \"ai-gateway-controller-7bc68ffc7c-pmmp6\" (UID: \"636400c6-9f4c-476d-964b-2bf5a279c586\") " pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.514250 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.514227 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636400c6-9f4c-476d-964b-2bf5a279c586-metrics-certs\") pod \"ai-gateway-controller-7bc68ffc7c-pmmp6\" (UID: \"636400c6-9f4c-476d-964b-2bf5a279c586\") " pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.565976 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.565934 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccskn\" (UniqueName: \"kubernetes.io/projected/636400c6-9f4c-476d-964b-2bf5a279c586-kube-api-access-ccskn\") pod \"ai-gateway-controller-7bc68ffc7c-pmmp6\" (UID: \"636400c6-9f4c-476d-964b-2bf5a279c586\") " pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.596715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.596690 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:42.965068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:42.964996 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6"] Nov 02 21:23:42.971729 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:23:42.971701 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636400c6_9f4c_476d_964b_2bf5a279c586.slice/crio-1d7b883c76a2732a52ee0982593d29ff62ee1c6d48a19f7ba94e18c09fc5460f WatchSource:0}: Error finding container 1d7b883c76a2732a52ee0982593d29ff62ee1c6d48a19f7ba94e18c09fc5460f: Status 404 returned error can't find the container with id 1d7b883c76a2732a52ee0982593d29ff62ee1c6d48a19f7ba94e18c09fc5460f Nov 02 21:23:43.086858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:43.086829 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" event={"ID":"636400c6-9f4c-476d-964b-2bf5a279c586","Type":"ContainerStarted","Data":"1d7b883c76a2732a52ee0982593d29ff62ee1c6d48a19f7ba94e18c09fc5460f"} Nov 02 21:23:44.156554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:44.156512 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:46.106462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:46.106431 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" event={"ID":"636400c6-9f4c-476d-964b-2bf5a279c586","Type":"ContainerStarted","Data":"c1b5d5d3273f6b3fe329b925273e4ee6cbb486596dd18731b48fb8db20b8b1b6"} Nov 02 21:23:46.106849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:46.106577 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:46.213749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:46.213696 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" podStartSLOduration=1.435578549 podStartE2EDuration="4.213682486s" totalImagesPullingTime="2.778103937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:46.212594743 +0000 UTC m=+655.768223079" watchObservedRunningTime="2026-11-02 21:23:46.213682486 +0000 UTC m=+655.769310822" Nov 02 21:23:57.113546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:57.113515 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" Nov 02 21:23:58.794866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:58.794835 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:59.296976 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:23:59.296931 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:00.778581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:00.778531 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:00.909129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:00.909099 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:02.268694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:02.268613 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:02.394676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:02.394644 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:10.613952 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:24:10.613920 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:24:27.074843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.074807 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:27.197407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.197378 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:27.585990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.585943 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp"] Nov 02 21:24:27.594208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.594187 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.643732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.643703 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c2dd3b-openshift-default-dockercfg-rvfbh\"" Nov 02 21:24:27.662035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.661980 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:27.693558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.693534 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp"] Nov 02 21:24:27.701357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701327 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701372 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701396 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701463 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701491 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmz6\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-kube-api-access-fxmz6\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701571 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701636 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701694 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.701879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.701833 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803315 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803346 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803383 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803417 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803433 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803456 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803477 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmz6\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-kube-api-access-fxmz6\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803499 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803763 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803811 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.803928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803846 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.804012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803948 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.804012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.803955 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.804207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.804188 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.805696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.805673 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.806049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.806032 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.857533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.857477 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmz6\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-kube-api-access-fxmz6\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.857740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.857722 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.908196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.907933 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:27.960319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:27.960273 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.087653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.087601 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.182654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.182572 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc"] Nov 02 21:24:28.187609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.187588 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.238615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.238574 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4d9ea0-openshift-default-dockercfg-vfpmv\"" Nov 02 21:24:28.241659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.241633 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc"] Nov 02 21:24:28.248993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.248959 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.308031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308004 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308042 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308065 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308082 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308148 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0216113-4b24-436b-801e-2c29e3bfb3c3-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308230 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psh4c\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-kube-api-access-psh4c\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308273 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308366 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.308449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.308392 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.385316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.385281 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.407231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.407202 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp"] Nov 02 21:24:28.409015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.408987 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409031 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409060 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409087 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409112 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0216113-4b24-436b-801e-2c29e3bfb3c3-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409140 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psh4c\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-kube-api-access-psh4c\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409179 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409257 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409276 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409497 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409599 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409620 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409874 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.409928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.409878 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0216113-4b24-436b-801e-2c29e3bfb3c3-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.411763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.411742 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.411906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.411885 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.413652 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:28.413631 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972c1f9c_7782_4cf3_9d79_9a7a7a0dfd95.slice/crio-af254792a4952c1cbfb08c3b5c35dc6a43407ad247efe039aa87dac7e2e0f98d WatchSource:0}: Error finding container af254792a4952c1cbfb08c3b5c35dc6a43407ad247efe039aa87dac7e2e0f98d: Status 404 returned error can't find the container with id af254792a4952c1cbfb08c3b5c35dc6a43407ad247efe039aa87dac7e2e0f98d Nov 02 21:24:28.415775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.415749 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:28.415838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.415810 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:28.415874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.415841 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:28.464138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.464112 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psh4c\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-kube-api-access-psh4c\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.464138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.464126 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.466712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.466689 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj"] Nov 02 21:24:28.470149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.470130 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.502248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.502217 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:28.511144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.511109 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.521002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.520932 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ef6b43-openshift-default-dockercfg-2pk6b\"" Nov 02 21:24:28.529140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.529089 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj"] Nov 02 21:24:28.612288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612257 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612297 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612323 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612463 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mggk\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-kube-api-access-8mggk\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612542 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612573 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff4fe47-15c6-456b-a5d4-431f95324537-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612664 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612751 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.612826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.612808 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.634506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.634472 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.713949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.713882 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.713949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.713937 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714014 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714061 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714095 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mggk\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-kube-api-access-8mggk\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714128 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714154 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff4fe47-15c6-456b-a5d4-431f95324537-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714193 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714253 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714371 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714475 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714576 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.714659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.714634 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.715096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.715045 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff4fe47-15c6-456b-a5d4-431f95324537-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.717133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.717114 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.717351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.717335 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.762439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.762413 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.769667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.769645 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.770282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.770260 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mggk\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-kube-api-access-8mggk\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.782206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.782185 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:28.887343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.887316 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.889320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.889298 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc"] Nov 02 21:24:28.895290 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:28.895264 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0216113_4b24_436b_801e_2c29e3bfb3c3.slice/crio-67a3a969590f0bdd6b41ecd9178a8e28800882ef40b902a6f6e583c2d381f085 WatchSource:0}: Error finding container 67a3a969590f0bdd6b41ecd9178a8e28800882ef40b902a6f6e583c2d381f085: Status 404 returned error can't find the container with id 67a3a969590f0bdd6b41ecd9178a8e28800882ef40b902a6f6e583c2d381f085 Nov 02 21:24:28.901013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.900952 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:28.901120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.901065 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:28.901120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:28.901105 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:29.033289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.033258 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:29.167200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.167165 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:29.173003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.172962 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj"] Nov 02 21:24:29.180817 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:29.180786 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff4fe47_15c6_456b_a5d4_431f95324537.slice/crio-070429a078300f632c50c332ad25a50a4830ee134d9fc4cdec93531c288ac645 WatchSource:0}: Error finding container 070429a078300f632c50c332ad25a50a4830ee134d9fc4cdec93531c288ac645: Status 404 returned error can't find the container with id 070429a078300f632c50c332ad25a50a4830ee134d9fc4cdec93531c288ac645 Nov 02 21:24:29.191115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.184744 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:29.191115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.184828 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:29.191115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.184873 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:29.305504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.305471 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:29.325171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.325066 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" event={"ID":"6ff4fe47-15c6-456b-a5d4-431f95324537","Type":"ContainerStarted","Data":"291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788"} Nov 02 21:24:29.325171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.325129 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" event={"ID":"6ff4fe47-15c6-456b-a5d4-431f95324537","Type":"ContainerStarted","Data":"070429a078300f632c50c332ad25a50a4830ee134d9fc4cdec93531c288ac645"} Nov 02 21:24:29.328318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.328288 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" event={"ID":"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95","Type":"ContainerStarted","Data":"02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e"} Nov 02 21:24:29.328461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.328332 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" event={"ID":"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95","Type":"ContainerStarted","Data":"af254792a4952c1cbfb08c3b5c35dc6a43407ad247efe039aa87dac7e2e0f98d"} Nov 02 21:24:29.332328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.332301 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" event={"ID":"d0216113-4b24-436b-801e-2c29e3bfb3c3","Type":"ContainerStarted","Data":"18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02"} Nov 02 21:24:29.332517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.332489 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" event={"ID":"d0216113-4b24-436b-801e-2c29e3bfb3c3","Type":"ContainerStarted","Data":"67a3a969590f0bdd6b41ecd9178a8e28800882ef40b902a6f6e583c2d381f085"} Nov 02 21:24:29.332666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.332649 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp"] Nov 02 21:24:29.337046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.337029 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.390097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.388864 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-dd6574-openshift-default-dockercfg-wcc2w\"" Nov 02 21:24:29.395686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.391824 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp"] Nov 02 21:24:29.420382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.420347 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430cd15-771a-4ace-85d9-6589bbd0ad57-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.420532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.420506 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.420605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.420589 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.420643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.420628 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tkc\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-kube-api-access-99tkc\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.420678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.420666 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.421146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.421119 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.421293 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.421269 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.421333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.421315 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.421367 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.421355 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.423078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.423058 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:29.441475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.441436 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podStartSLOduration=1.441424563 podStartE2EDuration="1.441424563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:29.438749518 +0000 UTC m=+698.994377879" watchObservedRunningTime="2026-11-02 21:24:29.441424563 +0000 UTC m=+698.997052958" Nov 02 21:24:29.503211 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.503133 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:29.508113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.508091 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:29.522623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522604 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.522706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522654 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.522706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522683 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.522706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522701 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.522835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522719 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.522835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522743 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430cd15-771a-4ace-85d9-6589bbd0ad57-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.522835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522775 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.523005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.522925 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.523153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.523127 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.523233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.523159 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.523323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.523146 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tkc\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-kube-api-access-99tkc\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.523483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.523319 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.523604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.523560 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.523676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.523644 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430cd15-771a-4ace-85d9-6589bbd0ad57-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.525536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.525510 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.525861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.525837 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.550744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.550685 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podStartSLOduration=1.550668908 podStartE2EDuration="1.550668908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:29.547419148 +0000 UTC m=+699.103047487" watchObservedRunningTime="2026-11-02 21:24:29.550668908 +0000 UTC m=+699.106297247" Nov 02 21:24:29.577040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.577001 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.577234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.577181 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tkc\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-kube-api-access-99tkc\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.651001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.650959 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:29.683576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.683538 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:29.764682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.764596 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podStartSLOduration=2.764582324 podStartE2EDuration="2.764582324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:29.763606543 +0000 UTC m=+699.319234881" watchObservedRunningTime="2026-11-02 21:24:29.764582324 +0000 UTC m=+699.320210661" Nov 02 21:24:29.782758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.782734 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:29.909210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.909175 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:29.914547 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:29.914523 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:30.029295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.029213 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp"] Nov 02 21:24:30.035456 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:30.035430 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8430cd15_771a_4ace_85d9_6589bbd0ad57.slice/crio-783e94d38dcdfcaeaa8a05a6ccbff022267b9cd6af701965f727866b7d0342e9 WatchSource:0}: Error finding container 783e94d38dcdfcaeaa8a05a6ccbff022267b9cd6af701965f727866b7d0342e9: Status 404 returned error can't find the container with id 783e94d38dcdfcaeaa8a05a6ccbff022267b9cd6af701965f727866b7d0342e9 Nov 02 21:24:30.037628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.037600 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:30.037722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.037661 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:30.037722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.037690 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:30.341034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.340995 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" event={"ID":"8430cd15-771a-4ace-85d9-6589bbd0ad57","Type":"ContainerStarted","Data":"80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7"} Nov 02 21:24:30.341487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.341062 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" event={"ID":"8430cd15-771a-4ace-85d9-6589bbd0ad57","Type":"ContainerStarted","Data":"783e94d38dcdfcaeaa8a05a6ccbff022267b9cd6af701965f727866b7d0342e9"} Nov 02 21:24:30.341570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.341545 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:30.341731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.341699 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:30.342492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.342471 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:24:30.342631 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.342615 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:24:30.453567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.453518 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" podStartSLOduration=1.4535056339999999 podStartE2EDuration="1.453505634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:30.450782057 +0000 UTC m=+700.006410413" watchObservedRunningTime="2026-11-02 21:24:30.453505634 +0000 UTC m=+700.009133971" Nov 02 21:24:30.652100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.652020 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:30.788695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:30.788667 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:31.347704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:31.347667 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:31.348654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:31.348636 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:31.656780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:31.656711 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:32.353950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:32.353918 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:32.354851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:32.354833 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:24:32.478257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:32.478223 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:32.631200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:32.631117 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:32.771384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:32.771358 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:33.128068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:33.128034 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:33.339861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:33.339829 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:33.467148 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:33.467080 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:33.735953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:33.735882 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:33.906183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:33.906154 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:34.027511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:34.027433 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:34.369778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:34.369746 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:34.534420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:34.534385 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:34.667057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:34.666961 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:34.967426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:34.967348 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:35.124523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:35.124456 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:35.305377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:35.305300 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:35.591682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:35.591655 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:35.769342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:35.769311 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:35.915560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:35.915488 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:37.688939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:37.688908 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:37.845863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:37.845835 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:38.093419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.093383 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq"] Nov 02 21:24:38.096815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.096795 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.146931 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.146903 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9a6cf2-dockercfg-x6qdp\"" Nov 02 21:24:38.147100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.147062 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9a6cf2\"" Nov 02 21:24:38.149348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.149321 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq"] Nov 02 21:24:38.211186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.211146 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7fd939e-a6b3-4579-b84d-c6b7da153315-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.211389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.211287 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvc72\" (UniqueName: \"kubernetes.io/projected/c7fd939e-a6b3-4579-b84d-c6b7da153315-kube-api-access-vvc72\") pod \"payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.312567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.312527 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvc72\" (UniqueName: \"kubernetes.io/projected/c7fd939e-a6b3-4579-b84d-c6b7da153315-kube-api-access-vvc72\") pod \"payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.312729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.312628 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7fd939e-a6b3-4579-b84d-c6b7da153315-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.313435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.313415 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7fd939e-a6b3-4579-b84d-c6b7da153315-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.368620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.368552 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvc72\" (UniqueName: \"kubernetes.io/projected/c7fd939e-a6b3-4579-b84d-c6b7da153315-kube-api-access-vvc72\") pod \"payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.409048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.409016 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:38.489063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.489014 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:38.637588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.637497 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:38.751634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.751591 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:38.780635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:38.780605 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq"] Nov 02 21:24:38.787324 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:38.787294 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fd939e_a6b3_4579_b84d_c6b7da153315.slice/crio-af8e00fa3d1ee75078a38060d830dcd0a502e9de56c1f695358ebb5972f9b673 WatchSource:0}: Error finding container af8e00fa3d1ee75078a38060d830dcd0a502e9de56c1f695358ebb5972f9b673: Status 404 returned error can't find the container with id af8e00fa3d1ee75078a38060d830dcd0a502e9de56c1f695358ebb5972f9b673 Nov 02 21:24:39.392562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:39.392526 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" event={"ID":"c7fd939e-a6b3-4579-b84d-c6b7da153315","Type":"ContainerStarted","Data":"af8e00fa3d1ee75078a38060d830dcd0a502e9de56c1f695358ebb5972f9b673"} Nov 02 21:24:40.618342 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:24:40.618300 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:24:42.245305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.245273 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:42.414625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.414588 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" event={"ID":"c7fd939e-a6b3-4579-b84d-c6b7da153315","Type":"ContainerStarted","Data":"b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148"} Nov 02 21:24:42.414815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.414674 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:42.414815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.414647 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:42.529799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.529650 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" podStartSLOduration=1.71280066 podStartE2EDuration="4.5296295s" totalImagesPullingTime="2.81682884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:42.521657061 +0000 UTC m=+712.077285402" watchObservedRunningTime="2026-11-02 21:24:42.5296295 +0000 UTC m=+712.085257862" Nov 02 21:24:42.820395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.820362 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw"] Nov 02 21:24:42.824077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.824057 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:42.887217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.887185 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw"] Nov 02 21:24:42.922457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.922429 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ef6b43-dockercfg-mpvnb\"" Nov 02 21:24:42.922457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.922439 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ef6b43\"" Nov 02 21:24:42.959515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.959486 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxjw\" (UniqueName: \"kubernetes.io/projected/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-kube-api-access-8lxjw\") pod \"payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:42.959650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:42.959528 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:43.060234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.060205 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxjw\" (UniqueName: \"kubernetes.io/projected/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-kube-api-access-8lxjw\") pod \"payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:43.060360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.060248 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:43.060897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.060878 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:43.116058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.115998 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxjw\" (UniqueName: \"kubernetes.io/projected/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-kube-api-access-8lxjw\") pod \"payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:43.136814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.136789 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:43.431524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.431398 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:43.507596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.507559 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw"] Nov 02 21:24:43.513910 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:43.513882 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c402d8_c4e6_4fb2_b43c_c360fa3da221.slice/crio-73f28beb482f1320b06684315719b4237173f0a69089d2ab728fe1e5a35b3b44 WatchSource:0}: Error finding container 73f28beb482f1320b06684315719b4237173f0a69089d2ab728fe1e5a35b3b44: Status 404 returned error can't find the container with id 73f28beb482f1320b06684315719b4237173f0a69089d2ab728fe1e5a35b3b44 Nov 02 21:24:43.590081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.590057 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:43.710006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:43.709895 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:44.426403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.426361 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" event={"ID":"a0c402d8-c4e6-4fb2-b43c-c360fa3da221","Type":"ContainerStarted","Data":"ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea"} Nov 02 21:24:44.426614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.426417 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" event={"ID":"a0c402d8-c4e6-4fb2-b43c-c360fa3da221","Type":"ContainerStarted","Data":"73f28beb482f1320b06684315719b4237173f0a69089d2ab728fe1e5a35b3b44"} Nov 02 21:24:44.426614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.426542 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:44.461581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.461557 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:44.535091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.535044 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" podStartSLOduration=2.5350303800000002 podStartE2EDuration="2.53503038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:44.533270551 +0000 UTC m=+714.088898887" watchObservedRunningTime="2026-11-02 21:24:44.53503038 +0000 UTC m=+714.090658717" Nov 02 21:24:44.700638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.700555 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:44.845714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.845686 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:44.994073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:44.993464 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:45.145933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.145901 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg"] Nov 02 21:24:45.149662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.149641 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.209275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.209233 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-4d9ea0\"" Nov 02 21:24:45.210873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.210848 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg"] Nov 02 21:24:45.255277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.255209 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-4d9ea0-dockercfg-k57sp\"" Nov 02 21:24:45.257344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.257323 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq"] Nov 02 21:24:45.260858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.260838 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.294932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.294908 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qm6s\" (UniqueName: \"kubernetes.io/projected/a9cffa72-fba9-446c-9780-130d7918e043-kube-api-access-9qm6s\") pod \"payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.295064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.294942 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cffa72-fba9-446c-9780-130d7918e043-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.314685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.314662 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq"] Nov 02 21:24:45.396058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.396023 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4428d-6985-4e19-abef-5e7ce965447a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.396218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.396089 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qm6s\" (UniqueName: \"kubernetes.io/projected/a9cffa72-fba9-446c-9780-130d7918e043-kube-api-access-9qm6s\") pod \"payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.396218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.396142 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cffa72-fba9-446c-9780-130d7918e043-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.396218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.396198 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn95c\" (UniqueName: \"kubernetes.io/projected/f0f4428d-6985-4e19-abef-5e7ce965447a-kube-api-access-mn95c\") pod \"payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.396874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.396852 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cffa72-fba9-446c-9780-130d7918e043-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.450119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.450088 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qm6s\" (UniqueName: \"kubernetes.io/projected/a9cffa72-fba9-446c-9780-130d7918e043-kube-api-access-9qm6s\") pod \"payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.462908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.462884 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:45.497275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.497246 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4428d-6985-4e19-abef-5e7ce965447a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.497402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.497325 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn95c\" (UniqueName: \"kubernetes.io/projected/f0f4428d-6985-4e19-abef-5e7ce965447a-kube-api-access-mn95c\") pod \"payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.497940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.497921 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4428d-6985-4e19-abef-5e7ce965447a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.551651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.551623 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn95c\" (UniqueName: \"kubernetes.io/projected/f0f4428d-6985-4e19-abef-5e7ce965447a-kube-api-access-mn95c\") pod \"payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.574432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.574404 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:45.836411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.836383 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg"] Nov 02 21:24:45.842683 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:45.842651 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cffa72_fba9_446c_9780_130d7918e043.slice/crio-6146bc32dfbd1114e3f92beb8eb03b00942c55ba38b1c3982c2502b6e9d3235d WatchSource:0}: Error finding container 6146bc32dfbd1114e3f92beb8eb03b00942c55ba38b1c3982c2502b6e9d3235d: Status 404 returned error can't find the container with id 6146bc32dfbd1114e3f92beb8eb03b00942c55ba38b1c3982c2502b6e9d3235d Nov 02 21:24:45.945397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:45.945370 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq"] Nov 02 21:24:45.950941 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:45.950916 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f4428d_6985_4e19_abef_5e7ce965447a.slice/crio-3c368a7fadd637ab37074839f686083825b8856fcc4bf705ab2843498bde48dc WatchSource:0}: Error finding container 3c368a7fadd637ab37074839f686083825b8856fcc4bf705ab2843498bde48dc: Status 404 returned error can't find the container with id 3c368a7fadd637ab37074839f686083825b8856fcc4bf705ab2843498bde48dc Nov 02 21:24:46.441437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.441385 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" event={"ID":"a9cffa72-fba9-446c-9780-130d7918e043","Type":"ContainerStarted","Data":"e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c"} Nov 02 21:24:46.441437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.441458 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" event={"ID":"a9cffa72-fba9-446c-9780-130d7918e043","Type":"ContainerStarted","Data":"6146bc32dfbd1114e3f92beb8eb03b00942c55ba38b1c3982c2502b6e9d3235d"} Nov 02 21:24:46.441720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.441628 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:46.443249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.443215 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" event={"ID":"f0f4428d-6985-4e19-abef-5e7ce965447a","Type":"ContainerStarted","Data":"d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518"} Nov 02 21:24:46.443361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.443264 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" event={"ID":"f0f4428d-6985-4e19-abef-5e7ce965447a","Type":"ContainerStarted","Data":"3c368a7fadd637ab37074839f686083825b8856fcc4bf705ab2843498bde48dc"} Nov 02 21:24:46.443361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.443297 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:46.551560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.551503 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" podStartSLOduration=1.5514910610000001 podStartE2EDuration="1.551491061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:46.549034301 +0000 UTC m=+716.104662640" watchObservedRunningTime="2026-11-02 21:24:46.551491061 +0000 UTC m=+716.107119397" Nov 02 21:24:46.659950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:46.659900 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" podStartSLOduration=1.659886278 podStartE2EDuration="1.659886278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:46.656684747 +0000 UTC m=+716.212313079" watchObservedRunningTime="2026-11-02 21:24:46.659886278 +0000 UTC m=+716.215514652" Nov 02 21:24:47.177533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.177496 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:47.347052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.347017 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:47.760086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.760049 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2"] Nov 02 21:24:47.763867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.763844 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:47.816195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.816157 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2"] Nov 02 21:24:47.860377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.860340 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c2dd3b\"" Nov 02 21:24:47.860377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.860369 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c2dd3b-dockercfg-brg4g\"" Nov 02 21:24:47.930167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.930127 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a550d-c276-4810-93e2-38b30676cbe3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:47.930344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:47.930241 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqnq\" (UniqueName: \"kubernetes.io/projected/ed5a550d-c276-4810-93e2-38b30676cbe3-kube-api-access-stqnq\") pod \"payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:48.031416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.031327 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a550d-c276-4810-93e2-38b30676cbe3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:48.031416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.031387 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqnq\" (UniqueName: \"kubernetes.io/projected/ed5a550d-c276-4810-93e2-38b30676cbe3-kube-api-access-stqnq\") pod \"payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:48.032141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.032115 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a550d-c276-4810-93e2-38b30676cbe3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:48.102990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.102108 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqnq\" (UniqueName: \"kubernetes.io/projected/ed5a550d-c276-4810-93e2-38b30676cbe3-kube-api-access-stqnq\") pod \"payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:48.104618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.104589 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:48.451678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.451644 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:48.492992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.477737 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2"] Nov 02 21:24:48.500489 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:48.500458 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5a550d_c276_4810_93e2_38b30676cbe3.slice/crio-39ca8f0e8d9fdb71554ad33ecfdb48516643179c9d1d9de53120fb9c2c6918f9 WatchSource:0}: Error finding container 39ca8f0e8d9fdb71554ad33ecfdb48516643179c9d1d9de53120fb9c2c6918f9: Status 404 returned error can't find the container with id 39ca8f0e8d9fdb71554ad33ecfdb48516643179c9d1d9de53120fb9c2c6918f9 Nov 02 21:24:48.604928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.604892 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:48.727219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:48.727148 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:49.497813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:49.497778 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" event={"ID":"ed5a550d-c276-4810-93e2-38b30676cbe3","Type":"ContainerStarted","Data":"555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b"} Nov 02 21:24:49.498266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:49.497833 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" event={"ID":"ed5a550d-c276-4810-93e2-38b30676cbe3","Type":"ContainerStarted","Data":"39ca8f0e8d9fdb71554ad33ecfdb48516643179c9d1d9de53120fb9c2c6918f9"} Nov 02 21:24:49.498266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:49.497988 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:24:49.609540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:49.609476 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" podStartSLOduration=2.609456946 podStartE2EDuration="2.609456946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:49.608804382 +0000 UTC m=+719.164432720" watchObservedRunningTime="2026-11-02 21:24:49.609456946 +0000 UTC m=+719.165085285" Nov 02 21:24:49.626807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:49.626779 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:49.807833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:49.807799 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:50.053945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.053908 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk"] Nov 02 21:24:50.057664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.057642 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.113032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.109483 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-284b8e-dockercfg-6m955\"" Nov 02 21:24:50.134150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.122554 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk"] Nov 02 21:24:50.151018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.150856 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4c566e-055c-494d-8299-d0871f6e6efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.151018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.150890 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vj2\" (UniqueName: \"kubernetes.io/projected/2d4c566e-055c-494d-8299-d0871f6e6efc-kube-api-access-w2vj2\") pod \"payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.156480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.156453 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-284b8e\"" Nov 02 21:24:50.251609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.251567 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4c566e-055c-494d-8299-d0871f6e6efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.251609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.251607 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vj2\" (UniqueName: \"kubernetes.io/projected/2d4c566e-055c-494d-8299-d0871f6e6efc-kube-api-access-w2vj2\") pod \"payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.252344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.252323 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4c566e-055c-494d-8299-d0871f6e6efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.321946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.321860 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vj2\" (UniqueName: \"kubernetes.io/projected/2d4c566e-055c-494d-8299-d0871f6e6efc-kube-api-access-w2vj2\") pod \"payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.408633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.408601 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:50.793210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:50.793127 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk"] Nov 02 21:24:50.797287 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:50.797257 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4c566e_055c_494d_8299_d0871f6e6efc.slice/crio-33f79ec77d020f2e1e0ad3e7a23b16b2188928a1489d2be6a72dc4382c9f64af WatchSource:0}: Error finding container 33f79ec77d020f2e1e0ad3e7a23b16b2188928a1489d2be6a72dc4382c9f64af: Status 404 returned error can't find the container with id 33f79ec77d020f2e1e0ad3e7a23b16b2188928a1489d2be6a72dc4382c9f64af Nov 02 21:24:51.512980 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:51.512941 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" event={"ID":"2d4c566e-055c-494d-8299-d0871f6e6efc","Type":"ContainerStarted","Data":"b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e"} Nov 02 21:24:51.513161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:51.513011 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" event={"ID":"2d4c566e-055c-494d-8299-d0871f6e6efc","Type":"ContainerStarted","Data":"33f79ec77d020f2e1e0ad3e7a23b16b2188928a1489d2be6a72dc4382c9f64af"} Nov 02 21:24:51.513226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:51.513155 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:24:53.258032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:53.257932 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:53.398579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:53.398529 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:53.423354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:53.423323 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 21:24:53.517592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:53.517509 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:53.531854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:53.531797 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" podStartSLOduration=3.53178491 podStartE2EDuration="3.53178491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:51.621248901 +0000 UTC m=+721.176877239" watchObservedRunningTime="2026-11-02 21:24:53.53178491 +0000 UTC m=+723.087413251" Nov 02 21:24:54.191319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.191284 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:54.349172 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.349141 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:54.601380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.601339 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s"] Nov 02 21:24:54.606055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.606034 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:54.657567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.657543 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s"] Nov 02 21:24:54.701328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.701285 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-dd6574\"" Nov 02 21:24:54.701328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.701354 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-dd6574-dockercfg-z8jd9\"" Nov 02 21:24:54.721310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.721277 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrmx7\" (UniqueName: \"kubernetes.io/projected/0f385b13-7059-40f1-876f-d9f0a0eaf369-kube-api-access-mrmx7\") pod \"payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:54.721496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.721470 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f385b13-7059-40f1-876f-d9f0a0eaf369-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:54.822566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.822526 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f385b13-7059-40f1-876f-d9f0a0eaf369-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:54.822748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.822655 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrmx7\" (UniqueName: \"kubernetes.io/projected/0f385b13-7059-40f1-876f-d9f0a0eaf369-kube-api-access-mrmx7\") pod \"payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:54.823297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.823274 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f385b13-7059-40f1-876f-d9f0a0eaf369-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:54.876552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.876478 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrmx7\" (UniqueName: \"kubernetes.io/projected/0f385b13-7059-40f1-876f-d9f0a0eaf369-kube-api-access-mrmx7\") pod \"payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:54.919740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:54.919709 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:55.296473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:55.296382 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s"] Nov 02 21:24:55.302931 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:24:55.302895 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f385b13_7059_40f1_876f_d9f0a0eaf369.slice/crio-68d43c514685dfaf4a4eaae4b9f1e150f790e8c969c5f09c41ec1bd0c1a8aeea WatchSource:0}: Error finding container 68d43c514685dfaf4a4eaae4b9f1e150f790e8c969c5f09c41ec1bd0c1a8aeea: Status 404 returned error can't find the container with id 68d43c514685dfaf4a4eaae4b9f1e150f790e8c969c5f09c41ec1bd0c1a8aeea Nov 02 21:24:55.434427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:55.434402 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:24:55.538294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:55.538260 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" event={"ID":"0f385b13-7059-40f1-876f-d9f0a0eaf369","Type":"ContainerStarted","Data":"9ef8454f649a9e2b517d0f086edf76e3757f2e96b99d607bac9eb640031ced14"} Nov 02 21:24:55.538485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:55.538322 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:24:55.538485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:55.538344 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" event={"ID":"0f385b13-7059-40f1-876f-d9f0a0eaf369","Type":"ContainerStarted","Data":"68d43c514685dfaf4a4eaae4b9f1e150f790e8c969c5f09c41ec1bd0c1a8aeea"} Nov 02 21:24:55.654280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:55.654226 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" podStartSLOduration=1.654212313 podStartE2EDuration="1.654212313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:55.650821014 +0000 UTC m=+725.206449350" watchObservedRunningTime="2026-11-02 21:24:55.654212313 +0000 UTC m=+725.209840650" Nov 02 21:24:56.547890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:56.547853 4441 generic.go:356] "Generic (PLEG): container finished" podID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerID="291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788" exitCode=0 Nov 02 21:24:56.548374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:56.547926 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" event={"ID":"6ff4fe47-15c6-456b-a5d4-431f95324537","Type":"ContainerDied","Data":"291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788"} Nov 02 21:24:56.548604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:56.548585 4441 scope.go:122] "RemoveContainer" containerID="291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788" Nov 02 21:24:56.550055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:56.550024 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:56.550139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:56.550107 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:56.550176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:56.550154 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:24:57.467793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:57.467766 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 21:24:57.468010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:57.467834 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 21:24:57.559087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:57.558564 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" event={"ID":"6ff4fe47-15c6-456b-a5d4-431f95324537","Type":"ContainerStarted","Data":"a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1"} Nov 02 21:24:57.783263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:57.783164 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:24:57.799384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:57.797132 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:24:57.799384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:57.797238 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 21:24:57.989008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:57.985458 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:58.787994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:58.783345 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:24:58.787994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:58.783418 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 21:24:58.967896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:58.967857 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:59.241993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.228275 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:59.406319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.406266 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw"] Nov 02 21:24:59.435126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.435090 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:24:59.446746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.446718 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw"] Nov 02 21:24:59.507045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.495262 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4e2b27\"" Nov 02 21:24:59.631996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.596077 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4e2b27-dockercfg-txjhk\"" Nov 02 21:24:59.656071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.615764 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6794621-aebc-4aac-9deb-e2e634c09275-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:24:59.656478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.656453 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhth\" (UniqueName: \"kubernetes.io/projected/f6794621-aebc-4aac-9deb-e2e634c09275-kube-api-access-kxhth\") pod \"payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:24:59.656765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.656746 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k"] Nov 02 21:24:59.666161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.665721 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:24:59.692991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.670158 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k"] Nov 02 21:24:59.757257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.757184 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6794621-aebc-4aac-9deb-e2e634c09275-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:24:59.757396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.757279 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhth\" (UniqueName: \"kubernetes.io/projected/f6794621-aebc-4aac-9deb-e2e634c09275-kube-api-access-kxhth\") pod \"payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:24:59.757897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.757874 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6794621-aebc-4aac-9deb-e2e634c09275-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:24:59.797371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.797332 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:24:59.809991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.797935 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 21:24:59.835996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.829858 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhth\" (UniqueName: \"kubernetes.io/projected/f6794621-aebc-4aac-9deb-e2e634c09275-kube-api-access-kxhth\") pod \"payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:24:59.858441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.858407 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854hr\" (UniqueName: \"kubernetes.io/projected/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-kube-api-access-854hr\") pod \"payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:24:59.858441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.858444 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:24:59.959677 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.959631 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854hr\" (UniqueName: \"kubernetes.io/projected/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-kube-api-access-854hr\") pod \"payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:24:59.959840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.959687 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:24:59.960686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:24:59.960655 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:25:00.045042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:00.044744 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854hr\" (UniqueName: \"kubernetes.io/projected/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-kube-api-access-854hr\") pod \"payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:25:00.086192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:00.086150 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:25:00.355323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:00.355193 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:25:00.523166 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:00.522155 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 21:25:00.667928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:00.666133 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw"] Nov 02 21:25:00.802991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:00.802360 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:25:00.802991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:00.802427 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 21:25:01.110451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.110408 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k"] Nov 02 21:25:01.159157 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:25:01.159108 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e9548b_d8f1_497a_94e6_1d6b72c8a571.slice/crio-505a29ea0fb3df9c048965f9b56cc21f508bdccf00674c9d9c0483b78c8db751 WatchSource:0}: Error finding container 505a29ea0fb3df9c048965f9b56cc21f508bdccf00674c9d9c0483b78c8db751: Status 404 returned error can't find the container with id 505a29ea0fb3df9c048965f9b56cc21f508bdccf00674c9d9c0483b78c8db751 Nov 02 21:25:01.349022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.348113 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:25:01.693560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.693516 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" event={"ID":"f6794621-aebc-4aac-9deb-e2e634c09275","Type":"ContainerStarted","Data":"c56e2cf8fc9879a5cee04eb0a49e6b4f60a892560fa3b3765e61d3bf04b0c521"} Nov 02 21:25:01.693885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.693862 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" event={"ID":"f6794621-aebc-4aac-9deb-e2e634c09275","Type":"ContainerStarted","Data":"79ae15f09fd06a84f84f94e4035fcab9280090a30f56ddcee2702f43ea157340"} Nov 02 21:25:01.698349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.698326 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:25:01.707239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.707214 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" event={"ID":"e9e9548b-d8f1-497a-94e6-1d6b72c8a571","Type":"ContainerStarted","Data":"3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2"} Nov 02 21:25:01.707412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.707393 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" event={"ID":"e9e9548b-d8f1-497a-94e6-1d6b72c8a571","Type":"ContainerStarted","Data":"505a29ea0fb3df9c048965f9b56cc21f508bdccf00674c9d9c0483b78c8db751"} Nov 02 21:25:01.708189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.708169 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:25:01.786425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.786386 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:25:01.787123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.786711 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 21:25:01.815472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.815400 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" podStartSLOduration=2.815380624 podStartE2EDuration="2.815380624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:01.813915929 +0000 UTC m=+731.369544264" watchObservedRunningTime="2026-11-02 21:25:01.815380624 +0000 UTC m=+731.371008962" Nov 02 21:25:01.926993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:01.926565 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" podStartSLOduration=2.9265455 podStartE2EDuration="2.9265455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:01.925930597 +0000 UTC m=+731.481558934" watchObservedRunningTime="2026-11-02 21:25:01.9265455 +0000 UTC m=+731.482173841" Nov 02 21:25:02.567993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:02.567477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:02.749535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:02.749329 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 21:25:02.756157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:02.756103 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:02.787870 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:02.787410 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:25:02.787870 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:02.787476 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 21:25:02.914599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:02.914507 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:03.789236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:03.789199 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:25:03.790475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:03.790451 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:25:06.551379 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.551306 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:25:06.751392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.751357 4441 generic.go:356] "Generic (PLEG): container finished" podID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerID="02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e" exitCode=0 Nov 02 21:25:06.751592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.751435 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" event={"ID":"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95","Type":"ContainerDied","Data":"02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e"} Nov 02 21:25:06.751881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.751855 4441 scope.go:122] "RemoveContainer" containerID="02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e" Nov 02 21:25:06.753236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.753201 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:06.753353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.753268 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:06.753353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.753299 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:06.753353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.753308 4441 generic.go:356] "Generic (PLEG): container finished" podID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerID="18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02" exitCode=0 Nov 02 21:25:06.753353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.753343 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" event={"ID":"d0216113-4b24-436b-801e-2c29e3bfb3c3","Type":"ContainerDied","Data":"18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02"} Nov 02 21:25:06.753785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.753715 4441 scope.go:122] "RemoveContainer" containerID="18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02" Nov 02 21:25:06.754853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.754829 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:06.754945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.754882 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:06.754945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:06.754915 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:07.144502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.144466 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:07.308991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.307996 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:07.422642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.422548 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:07.761180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.761101 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" event={"ID":"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95","Type":"ContainerStarted","Data":"63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e"} Nov 02 21:25:07.763163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.763137 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" event={"ID":"d0216113-4b24-436b-801e-2c29e3bfb3c3","Type":"ContainerStarted","Data":"0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41"} Nov 02 21:25:07.921045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.920528 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:25:07.921045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.920566 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:25:07.921045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:07.920615 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 21:25:08.504004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:08.503364 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:25:08.504004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:08.503494 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 21:25:08.504004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:08.503538 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 21:25:08.921712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:08.921671 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:25:08.922364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:08.922334 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 21:25:09.503798 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:09.503760 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 21:25:09.504136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:09.504105 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 21:25:09.922676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:09.922642 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:25:09.923306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:09.923276 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 21:25:10.507993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:10.507306 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 21:25:10.507993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:10.507365 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 21:25:10.653695 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:25:10.653658 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:25:10.911988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:10.908630 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:25:10.911988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:10.908689 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 21:25:11.506409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:11.506370 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 21:25:11.507040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:11.507009 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 21:25:11.740953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:11.740925 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:11.910567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:11.910522 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:25:11.910879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:11.910837 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 21:25:11.920602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:11.920579 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:12.183486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:12.183400 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:12.504050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:12.503387 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 21:25:12.504050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:12.503451 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 21:25:12.920253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:12.920215 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:25:12.920839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:12.920804 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 21:25:13.506709 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.506672 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:25:13.507008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.506980 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:25:13.507738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.507723 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:25:13.729785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.729758 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 21:25:13.729950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.729875 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 21:25:13.913284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.913244 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:25:13.913645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.913624 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:25:13.914473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:13.914450 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 21:25:16.391862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:16.391829 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:20.960011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:20.959947 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:22.813999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:22.813954 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:23.390993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:23.390869 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:23.518949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:23.518912 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:23.642186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:23.642105 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:25.845059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:25.845026 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:26.894955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:26.894912 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v"] Nov 02 21:25:26.898588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:26.898562 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:26.953243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:26.952912 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v"] Nov 02 21:25:26.995415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:26.995385 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"openshift-service-ca.crt\"" Nov 02 21:25:26.995579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:26.995530 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"default-dockercfg-fgsj4\"" Nov 02 21:25:26.996408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:26.996383 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"kube-root-ca.crt\"" Nov 02 21:25:26.996408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:26.996402 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"simulator-w5-ef6b43-kserve-self-signed-certs\"" Nov 02 21:25:27.010873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.010853 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:27.020383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.020360 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpgcw\" (UniqueName: \"kubernetes.io/projected/8d6e43ff-e946-4b43-be35-672d616aac21-kube-api-access-dpgcw\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.020462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.020421 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-tmp-dir\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.020517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.020487 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-dshm\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.020559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.020526 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-home\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.020625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.020608 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-model-cache\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.020666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.020649 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6e43ff-e946-4b43-be35-672d616aac21-tls-certs\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.121616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.121581 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-tmp-dir\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.121819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.121626 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-dshm\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.121819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.121679 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-home\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.121819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.121732 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-model-cache\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.121819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.121772 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6e43ff-e946-4b43-be35-672d616aac21-tls-certs\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.122070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.121835 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpgcw\" (UniqueName: \"kubernetes.io/projected/8d6e43ff-e946-4b43-be35-672d616aac21-kube-api-access-dpgcw\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.122070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.121961 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-tmp-dir\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.122296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.122251 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-home\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.122437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.122383 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-model-cache\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.125056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.125025 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-dshm\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.125430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.125406 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6e43ff-e946-4b43-be35-672d616aac21-tls-certs\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.176983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.176904 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpgcw\" (UniqueName: \"kubernetes.io/projected/8d6e43ff-e946-4b43-be35-672d616aac21-kube-api-access-dpgcw\") pod \"simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.213067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.213041 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:27.591240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.591211 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v"] Nov 02 21:25:27.596646 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:25:27.596620 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d6e43ff_e946_4b43_be35_672d616aac21.slice/crio-705d978d4ae73c59594a336a812804e52363f02de51875ee19fd52832dbb20da WatchSource:0}: Error finding container 705d978d4ae73c59594a336a812804e52363f02de51875ee19fd52832dbb20da: Status 404 returned error can't find the container with id 705d978d4ae73c59594a336a812804e52363f02de51875ee19fd52832dbb20da Nov 02 21:25:27.953244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.953153 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" event={"ID":"8d6e43ff-e946-4b43-be35-672d616aac21","Type":"ContainerStarted","Data":"8c444be9ba9abc2c7f2244f7ba216f40aba1f710d1461141a2e98f1d6495c029"} Nov 02 21:25:27.953244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:27.953212 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" event={"ID":"8d6e43ff-e946-4b43-be35-672d616aac21","Type":"ContainerStarted","Data":"705d978d4ae73c59594a336a812804e52363f02de51875ee19fd52832dbb20da"} Nov 02 21:25:28.063525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.063456 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podStartSLOduration=2.063437001 podStartE2EDuration="2.063437001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:28.061043112 +0000 UTC m=+757.616671476" watchObservedRunningTime="2026-11-02 21:25:28.063437001 +0000 UTC m=+757.619065339" Nov 02 21:25:28.094746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.094706 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:28.213402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.213327 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:28.680859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.680790 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:28.812784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.812747 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:28.933565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.933484 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:28.961442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.961400 4441 generic.go:356] "Generic (PLEG): container finished" podID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerID="80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7" exitCode=0 Nov 02 21:25:28.961909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.961877 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" event={"ID":"8430cd15-771a-4ace-85d9-6589bbd0ad57","Type":"ContainerDied","Data":"80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7"} Nov 02 21:25:28.962242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.962223 4441 scope.go:122] "RemoveContainer" containerID="80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7" Nov 02 21:25:28.963457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.963430 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:28.963569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.963510 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:28.963569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:28.963556 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:25:29.996357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:29.996315 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" event={"ID":"8430cd15-771a-4ace-85d9-6589bbd0ad57","Type":"ContainerStarted","Data":"93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384"} Nov 02 21:25:30.652010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:30.651932 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:25:30.652914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:30.652891 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 21:25:30.653023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:30.652943 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 21:25:30.801180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:30.801148 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:31.651624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:31.651587 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 21:25:31.652042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:31.651642 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 21:25:32.354398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:32.354360 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:25:32.651447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:32.651372 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 21:25:32.651628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:32.651452 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 21:25:33.268799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:33.268761 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:33.651804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:33.651757 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 21:25:33.652070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:33.651823 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 21:25:33.934749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:33.934661 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:34.195262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:34.195176 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:34.329930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:34.329894 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:34.652295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:34.652260 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 21:25:34.652463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:34.652310 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 21:25:35.656030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:35.655999 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:25:35.657161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:35.657138 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:25:35.683652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:35.683625 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:37.213703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:37.213669 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:37.226182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:37.226156 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:38.068150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:38.068120 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:25:39.484574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:39.484514 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:40.046723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:40.046690 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:40.178224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:40.178176 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:40.309514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:40.309487 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:40.614881 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:25:40.614798 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:25:40.812860 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:40.812829 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:44.771414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:44.771316 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:45.401936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:45.401906 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:47.520048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:47.520020 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:48.110099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:48.110067 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:48.240992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:48.240943 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:48.378683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:48.378580 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:48.986620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:48.986589 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:50.263478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:50.263443 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:54.692605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:54.692569 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:54.999466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:54.999354 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:55.304162 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:55.304090 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:55.425530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:55.425503 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:56.309446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.309406 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6"] Nov 02 21:25:56.314227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.314201 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.391509 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.391480 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6"] Nov 02 21:25:56.418494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.418439 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"default-dockercfg-lnkl5\"" Nov 02 21:25:56.418494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.418463 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"openshift-service-ca.crt\"" Nov 02 21:25:56.418494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.418476 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"kube-root-ca.crt\"" Nov 02 21:25:56.434891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.434861 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4k8\" (UniqueName: \"kubernetes.io/projected/d07e2fd5-f023-4507-aa4a-854a9b490a1d-kube-api-access-xl4k8\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.435020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.434921 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-dshm\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.435076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.435039 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tls-certs\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.435132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.435096 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-model-cache\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.435132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.435119 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-home\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.435227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.435178 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tmp-dir\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.460708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.460644 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"simulator-w1-c2dd3b-kserve-self-signed-certs\"" Nov 02 21:25:56.469468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.469442 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:56.536024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.535992 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-dshm\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536050 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tls-certs\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536097 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-model-cache\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536117 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-home\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536131 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tmp-dir\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536179 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4k8\" (UniqueName: \"kubernetes.io/projected/d07e2fd5-f023-4507-aa4a-854a9b490a1d-kube-api-access-xl4k8\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536556 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536511 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-model-cache\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536560 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-home\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.536674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.536622 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tmp-dir\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.538894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.538843 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-dshm\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.539241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.539223 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tls-certs\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.592287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.592231 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4k8\" (UniqueName: \"kubernetes.io/projected/d07e2fd5-f023-4507-aa4a-854a9b490a1d-kube-api-access-xl4k8\") pod \"simulator-w1-c2dd3b-kserve-67b97458d4-dngn6\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:56.628260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:56.628222 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:57.001567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:57.001487 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6"] Nov 02 21:25:57.007526 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:25:57.007496 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd07e2fd5_f023_4507_aa4a_854a9b490a1d.slice/crio-66d4bd36dd9ede51662f56105446b8291894bb175016ea10ca9f990a49f03a1e WatchSource:0}: Error finding container 66d4bd36dd9ede51662f56105446b8291894bb175016ea10ca9f990a49f03a1e: Status 404 returned error can't find the container with id 66d4bd36dd9ede51662f56105446b8291894bb175016ea10ca9f990a49f03a1e Nov 02 21:25:57.179197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:57.179154 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" event={"ID":"d07e2fd5-f023-4507-aa4a-854a9b490a1d","Type":"ContainerStarted","Data":"25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836"} Nov 02 21:25:57.179197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:57.179203 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" event={"ID":"d07e2fd5-f023-4507-aa4a-854a9b490a1d","Type":"ContainerStarted","Data":"66d4bd36dd9ede51662f56105446b8291894bb175016ea10ca9f990a49f03a1e"} Nov 02 21:25:57.286590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:57.286481 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podStartSLOduration=1.286466162 podStartE2EDuration="1.286466162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:57.286229395 +0000 UTC m=+786.841857746" watchObservedRunningTime="2026-11-02 21:25:57.286466162 +0000 UTC m=+786.842094498" Nov 02 21:25:57.628735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:57.628702 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:25:58.816951 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:58.816899 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:59.394010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:59.393956 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:59.527009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:59.526956 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:59.659337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:59.659241 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:59.801035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:25:59.801005 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:04.394583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:04.394551 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:06.436666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:06.436628 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:06.629045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:06.629009 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:26:06.641403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:06.641377 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:26:07.257307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:07.257278 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:26:08.887806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:08.887750 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:10.614481 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:26:10.614443 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:26:13.399175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:13.399116 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:18.032705 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:18.032604 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:20.891982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.891940 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q"] Nov 02 21:26:20.894948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.894925 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:20.975001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.974951 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:20.990240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.990207 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-home\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:20.990363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.990325 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5s7r\" (UniqueName: \"kubernetes.io/projected/f9377f9b-5bcc-4347-a55d-8b38059f19ca-kube-api-access-m5s7r\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:20.990415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.990364 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-model-cache\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:20.990470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.990438 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-dshm\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:20.990525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.990499 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tmp-dir\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:20.990567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.990537 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tls-certs\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:20.991653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.991620 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"kube-root-ca.crt\"" Nov 02 21:26:20.991653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.991634 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"openshift-service-ca.crt\"" Nov 02 21:26:20.991820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.991696 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"default-dockercfg-44bw2\"" Nov 02 21:26:20.991957 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.991940 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"simulator-w4-4e2b27-kserve-self-signed-certs\"" Nov 02 21:26:20.994651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:20.994634 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q"] Nov 02 21:26:21.091065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091030 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-dshm\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091092 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tmp-dir\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tls-certs\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091150 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-home\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091212 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5s7r\" (UniqueName: \"kubernetes.io/projected/f9377f9b-5bcc-4347-a55d-8b38059f19ca-kube-api-access-m5s7r\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091248 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-model-cache\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091579 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tmp-dir\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091631 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-model-cache\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.091668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.091636 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-home\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.093498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.093475 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-dshm\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.093612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.093594 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tls-certs\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.144664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.144606 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5s7r\" (UniqueName: \"kubernetes.io/projected/f9377f9b-5bcc-4347-a55d-8b38059f19ca-kube-api-access-m5s7r\") pod \"simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.210644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.210620 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:21.581454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:21.581423 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q"] Nov 02 21:26:21.587838 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:26:21.587815 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9377f9b_5bcc_4347_a55d_8b38059f19ca.slice/crio-61966cd8f87e378b14d33e5f0da1eef00238bb43ce49a7a34626d982d2fa8d11 WatchSource:0}: Error finding container 61966cd8f87e378b14d33e5f0da1eef00238bb43ce49a7a34626d982d2fa8d11: Status 404 returned error can't find the container with id 61966cd8f87e378b14d33e5f0da1eef00238bb43ce49a7a34626d982d2fa8d11 Nov 02 21:26:22.339931 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:22.339897 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" event={"ID":"f9377f9b-5bcc-4347-a55d-8b38059f19ca","Type":"ContainerStarted","Data":"c714d4d170d086058ed71dfb45aec03e92592384c7d8867773f02ccce109448e"} Nov 02 21:26:22.340337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:22.339940 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" event={"ID":"f9377f9b-5bcc-4347-a55d-8b38059f19ca","Type":"ContainerStarted","Data":"61966cd8f87e378b14d33e5f0da1eef00238bb43ce49a7a34626d982d2fa8d11"} Nov 02 21:26:22.502276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:22.502211 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:23.210903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:23.210872 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:25.932443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:25.932405 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:26.980186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:26.980133 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:31.211507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:31.211467 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:31.224585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:31.224557 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:31.335055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:31.334985 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podStartSLOduration=11.334946581 podStartE2EDuration="11.334946581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:22.44886216 +0000 UTC m=+812.004490498" watchObservedRunningTime="2026-11-02 21:26:31.334946581 +0000 UTC m=+820.890574922" Nov 02 21:26:31.411781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:31.411743 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:26:31.496334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:31.496239 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:34.537389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.537352 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp"] Nov 02 21:26:34.541888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.541863 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.592691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.592656 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp"] Nov 02 21:26:34.612503 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.612469 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dd6574\"/\"openshift-service-ca.crt\"" Nov 02 21:26:34.622194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.622163 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:34.632857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.632832 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-tmp-dir\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.632989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.632868 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-model-cache\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.632989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.632887 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvql\" (UniqueName: \"kubernetes.io/projected/17df70f7-9cbe-452b-96ff-0edad519ec94-kube-api-access-kmvql\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.633158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.633139 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17df70f7-9cbe-452b-96ff-0edad519ec94-tls-certs\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.633215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.633186 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-dshm\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.633279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.633262 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-home\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.636652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.636633 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dd6574\"/\"kube-root-ca.crt\"" Nov 02 21:26:34.636768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.636745 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dd6574\"/\"simulator-w0-dd6574-kserve-self-signed-certs\"" Nov 02 21:26:34.638489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.638471 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dd6574\"/\"default-dockercfg-sm622\"" Nov 02 21:26:34.734259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734225 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-tmp-dir\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734265 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-model-cache\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734286 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvql\" (UniqueName: \"kubernetes.io/projected/17df70f7-9cbe-452b-96ff-0edad519ec94-kube-api-access-kmvql\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734384 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17df70f7-9cbe-452b-96ff-0edad519ec94-tls-certs\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734414 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-dshm\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734437 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-home\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734691 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-model-cache\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734727 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-tmp-dir\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.734837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.734821 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-home\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.736671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.736648 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-dshm\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.736926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.736904 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17df70f7-9cbe-452b-96ff-0edad519ec94-tls-certs\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.787811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.787756 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvql\" (UniqueName: \"kubernetes.io/projected/17df70f7-9cbe-452b-96ff-0edad519ec94-kube-api-access-kmvql\") pod \"simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:34.853918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:34.853891 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:35.223532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:35.223497 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp"] Nov 02 21:26:35.228947 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:26:35.228919 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17df70f7_9cbe_452b_96ff_0edad519ec94.slice/crio-39843b96f8dc2f3dc0b6ca608f9ca2e661c446c58557971a26fb044082b25c4d WatchSource:0}: Error finding container 39843b96f8dc2f3dc0b6ca608f9ca2e661c446c58557971a26fb044082b25c4d: Status 404 returned error can't find the container with id 39843b96f8dc2f3dc0b6ca608f9ca2e661c446c58557971a26fb044082b25c4d Nov 02 21:26:35.428629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:35.428587 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" event={"ID":"17df70f7-9cbe-452b-96ff-0edad519ec94","Type":"ContainerStarted","Data":"0b0f7c4cd37393a5cf5202b96d6e4ec9537b51fb057a4be3f0ed34b77d071eee"} Nov 02 21:26:35.428629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:35.428633 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" event={"ID":"17df70f7-9cbe-452b-96ff-0edad519ec94","Type":"ContainerStarted","Data":"39843b96f8dc2f3dc0b6ca608f9ca2e661c446c58557971a26fb044082b25c4d"} Nov 02 21:26:35.536894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:35.536802 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podStartSLOduration=1.536788021 podStartE2EDuration="1.536788021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:35.536130611 +0000 UTC m=+825.091758950" watchObservedRunningTime="2026-11-02 21:26:35.536788021 +0000 UTC m=+825.092416357" Nov 02 21:26:35.854257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:35.854227 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:35.985855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:35.985827 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:38.747150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:38.747118 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:40.473888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:40.473851 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:40.614023 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:26:40.613987 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:26:44.855039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:44.855006 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:44.868107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:44.868079 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:44.999043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:44.998993 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:45.125906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:45.125805 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:45.524448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:45.524369 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:26:49.179644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:49.179609 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:49.300066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:49.300036 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:49.528912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:49.528843 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:53.678101 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.678064 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9"] Nov 02 21:26:53.681539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.681512 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.732831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.732802 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9"] Nov 02 21:26:53.763571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.763542 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"openshift-service-ca.crt\"" Nov 02 21:26:53.769273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.769249 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:53.776443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.776425 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"kube-root-ca.crt\"" Nov 02 21:26:53.776586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.776565 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"premium-w2-4d9ea0-kserve-self-signed-certs\"" Nov 02 21:26:53.778262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.778244 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"default-dockercfg-fpf4h\"" Nov 02 21:26:53.842508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.842466 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-model-cache\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.842803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.842780 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4b6r\" (UniqueName: \"kubernetes.io/projected/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-kube-api-access-s4b6r\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.842945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.842926 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-home\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.843132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.843113 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tls-certs\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.843317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.843299 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tmp-dir\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.843626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.843606 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-dshm\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.944765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.944680 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tmp-dir\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.944916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.944771 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-dshm\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.944916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.944866 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-model-cache\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.944916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.944895 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4b6r\" (UniqueName: \"kubernetes.io/projected/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-kube-api-access-s4b6r\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.945133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.944933 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-home\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.945133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.945007 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tls-certs\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.945243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.945137 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tmp-dir\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.945308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.945264 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-model-cache\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.945366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.945355 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-home\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.947084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.947065 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-dshm\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.947432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.947412 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tls-certs\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:53.998502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:53.998477 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4b6r\" (UniqueName: \"kubernetes.io/projected/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-kube-api-access-s4b6r\") pod \"premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:54.061411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:54.061380 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:54.293137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:54.293052 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:54.661881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:54.661845 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9"] Nov 02 21:26:54.667784 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:26:54.667759 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod522cb6cf_1fa2_4eee_8f06_f1e5fc55594f.slice/crio-70417f99de0ca4c42b3aa89be2d610e6a47889654f9e9c647d52034eada09308 WatchSource:0}: Error finding container 70417f99de0ca4c42b3aa89be2d610e6a47889654f9e9c647d52034eada09308: Status 404 returned error can't find the container with id 70417f99de0ca4c42b3aa89be2d610e6a47889654f9e9c647d52034eada09308 Nov 02 21:26:55.578586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:55.578549 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" event={"ID":"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f","Type":"ContainerStarted","Data":"e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06"} Nov 02 21:26:55.578960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:55.578595 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" event={"ID":"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f","Type":"ContainerStarted","Data":"70417f99de0ca4c42b3aa89be2d610e6a47889654f9e9c647d52034eada09308"} Nov 02 21:26:55.686091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:55.686029 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podStartSLOduration=2.686015281 podStartE2EDuration="2.686015281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:55.685316187 +0000 UTC m=+845.240944523" watchObservedRunningTime="2026-11-02 21:26:55.686015281 +0000 UTC m=+845.241643617" Nov 02 21:26:56.294164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:56.294130 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:26:58.580395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:26:58.580364 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:03.082451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:03.082367 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:04.294178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:04.294142 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:27:04.307298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:04.307270 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:27:04.653174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:04.653144 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:27:07.580367 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:07.580329 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:10.616362 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:27:10.616319 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:27:12.083213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:12.083160 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:16.582331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:16.582296 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:21.006094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:21.006058 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:21.122215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:21.122185 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:25.614499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:25.614465 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:27.877915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.877883 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj"] Nov 02 21:27:27.881004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.880981 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:27.968825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.968797 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:27.977697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.977669 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"kube-root-ca.crt\"" Nov 02 21:27:27.977796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.977716 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"premium-w6-284b8e-kserve-self-signed-certs\"" Nov 02 21:27:27.977796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.977717 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"openshift-service-ca.crt\"" Nov 02 21:27:27.977796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.977734 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"default-dockercfg-kqqs4\"" Nov 02 21:27:27.980728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:27.980711 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj"] Nov 02 21:27:28.000173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.000146 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-dshm\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.000297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.000178 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-tmp-dir\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.000297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.000267 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-model-cache\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.000417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.000380 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-home\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.000417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.000407 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4jz\" (UniqueName: \"kubernetes.io/projected/93c7eea9-bba3-4e14-9702-3c837a91d97d-kube-api-access-gk4jz\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.000531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.000438 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93c7eea9-bba3-4e14-9702-3c837a91d97d-tls-certs\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.101847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.101813 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-dshm\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.101847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.101852 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-tmp-dir\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.102123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.101896 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-model-cache\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.102123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.102018 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-home\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.102123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.102056 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4jz\" (UniqueName: \"kubernetes.io/projected/93c7eea9-bba3-4e14-9702-3c837a91d97d-kube-api-access-gk4jz\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.102123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.102113 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93c7eea9-bba3-4e14-9702-3c837a91d97d-tls-certs\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.102342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.102317 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-model-cache\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.102504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.102477 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-tmp-dir\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.102580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.102513 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-home\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.104247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.104213 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-dshm\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.104547 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.104527 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93c7eea9-bba3-4e14-9702-3c837a91d97d-tls-certs\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.162391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.162336 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4jz\" (UniqueName: \"kubernetes.io/projected/93c7eea9-bba3-4e14-9702-3c837a91d97d-kube-api-access-gk4jz\") pod \"premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.191986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.191944 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:28.560459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.560431 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj"] Nov 02 21:27:28.566515 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:27:28.566487 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93c7eea9_bba3_4e14_9702_3c837a91d97d.slice/crio-ff0f3b92fe10e9317f91f539a0eec3c6276c8926b12b5019ff8c59368ff27992 WatchSource:0}: Error finding container ff0f3b92fe10e9317f91f539a0eec3c6276c8926b12b5019ff8c59368ff27992: Status 404 returned error can't find the container with id ff0f3b92fe10e9317f91f539a0eec3c6276c8926b12b5019ff8c59368ff27992 Nov 02 21:27:28.795734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.795705 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" event={"ID":"93c7eea9-bba3-4e14-9702-3c837a91d97d","Type":"ContainerStarted","Data":"a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18"} Nov 02 21:27:28.795887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.795745 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" event={"ID":"93c7eea9-bba3-4e14-9702-3c837a91d97d","Type":"ContainerStarted","Data":"ff0f3b92fe10e9317f91f539a0eec3c6276c8926b12b5019ff8c59368ff27992"} Nov 02 21:27:28.905326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:28.905226 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podStartSLOduration=1.905214012 podStartE2EDuration="1.905214012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:28.903649457 +0000 UTC m=+878.459277795" watchObservedRunningTime="2026-11-02 21:27:28.905214012 +0000 UTC m=+878.460842349" Nov 02 21:27:29.192143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:29.192060 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:30.178946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:30.178912 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:34.638851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:34.638806 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:38.192272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:38.192233 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:38.205788 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:38.205762 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:38.877726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:38.877691 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:27:39.128259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:39.128165 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:40.614209 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:27:40.614172 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:27:41.521756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:41.521721 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:43.699173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:43.699133 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:45.458402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:45.458361 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:48.195051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:48.195009 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:50.671504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:50.671473 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:27:50.674439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:50.674415 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:27:50.682143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:50.682119 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:27:50.685392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:50.685368 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:27:50.694883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:50.694863 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:27:50.697427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:50.697404 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:27:51.614034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:51.613462 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:51.738824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:51.738785 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:52.824881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:52.824844 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:55.253419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.253383 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm"] Nov 02 21:27:55.257613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.257590 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.338199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.338157 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:55.354652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.354626 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dd6574\"/\"premium-w0-dd6574-kserve-self-signed-certs\"" Nov 02 21:27:55.358023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.357991 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm"] Nov 02 21:27:55.409706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.409667 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f454f72-4073-4bbe-9f10-c11ba9df402f-tls-certs\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.410035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.410012 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-home\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.410345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.410324 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-model-cache\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.410555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.410538 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-tmp-dir\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.413721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.413455 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5knx\" (UniqueName: \"kubernetes.io/projected/8f454f72-4073-4bbe-9f10-c11ba9df402f-kube-api-access-w5knx\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.414072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.414029 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-dshm\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.515854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.515775 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-dshm\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.515854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.515839 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f454f72-4073-4bbe-9f10-c11ba9df402f-tls-certs\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.516039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.515894 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-home\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.516039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.515961 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-model-cache\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.516039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.516016 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-tmp-dir\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.516147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.516055 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5knx\" (UniqueName: \"kubernetes.io/projected/8f454f72-4073-4bbe-9f10-c11ba9df402f-kube-api-access-w5knx\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.516542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.516516 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-model-cache\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.516657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.516600 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-home\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.516831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.516812 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-tmp-dir\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.518497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.518472 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-dshm\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.518583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.518551 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f454f72-4073-4bbe-9f10-c11ba9df402f-tls-certs\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.570138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.570114 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5knx\" (UniqueName: \"kubernetes.io/projected/8f454f72-4073-4bbe-9f10-c11ba9df402f-kube-api-access-w5knx\") pod \"premium-w0-dd6574-kserve-68794fdc76-5jqpm\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.571899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.571869 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:55.961217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.961185 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm"] Nov 02 21:27:55.967163 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:27:55.967132 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f454f72_4073_4bbe_9f10_c11ba9df402f.slice/crio-341178be947056c4871eb28ea286bbebf5cadcb8eab1b44f8d81c5514c4a9686 WatchSource:0}: Error finding container 341178be947056c4871eb28ea286bbebf5cadcb8eab1b44f8d81c5514c4a9686: Status 404 returned error can't find the container with id 341178be947056c4871eb28ea286bbebf5cadcb8eab1b44f8d81c5514c4a9686 Nov 02 21:27:55.984892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:55.984866 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" event={"ID":"8f454f72-4073-4bbe-9f10-c11ba9df402f","Type":"ContainerStarted","Data":"341178be947056c4871eb28ea286bbebf5cadcb8eab1b44f8d81c5514c4a9686"} Nov 02 21:27:56.992984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:56.992941 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" event={"ID":"8f454f72-4073-4bbe-9f10-c11ba9df402f","Type":"ContainerStarted","Data":"646ad6d52da12e1def53b5ef30e82d13c415d1aeded8b81990c5c31e1a1740e3"} Nov 02 21:27:57.103315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:57.103266 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podStartSLOduration=2.103251183 podStartE2EDuration="2.103251183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:57.100459289 +0000 UTC m=+906.656087625" watchObservedRunningTime="2026-11-02 21:27:57.103251183 +0000 UTC m=+906.658879520" Nov 02 21:27:57.383064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:57.383031 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:57.572475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:57.572440 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:27:59.773664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.773625 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc"] Nov 02 21:27:59.777896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.777871 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.839703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.839671 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc"] Nov 02 21:27:59.865252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.865221 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.865419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.865282 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5qb\" (UniqueName: \"kubernetes.io/projected/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-kube-api-access-bn5qb\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.865478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.865411 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.873564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.873537 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:27:59.873564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.873539 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:27:59.873761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.873546 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:27:59.875097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.875077 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:27:59.966402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.966378 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.966546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.966421 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5qb\" (UniqueName: \"kubernetes.io/projected/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-kube-api-access-bn5qb\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.966546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.966462 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.967066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.967044 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:27:59.968817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:27:59.968794 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:28:00.020028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:00.019997 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5qb\" (UniqueName: \"kubernetes.io/projected/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-kube-api-access-bn5qb\") pod \"maas-api-revoke-sub-1ff1daeb-mx9kc\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:28:00.091495 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:00.091464 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:28:00.465028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:00.464940 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc"] Nov 02 21:28:00.471764 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:28:00.471719 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69c6c9d_23d3_4be2_ad09_158a6cf8dbda.slice/crio-fc6286c4dd9535f2dc44727fc082d8ed46494d124855beade643ddf6bd66c786 WatchSource:0}: Error finding container fc6286c4dd9535f2dc44727fc082d8ed46494d124855beade643ddf6bd66c786: Status 404 returned error can't find the container with id fc6286c4dd9535f2dc44727fc082d8ed46494d124855beade643ddf6bd66c786 Nov 02 21:28:00.474307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:00.474281 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:28:01.019768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:01.019734 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" event={"ID":"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda","Type":"ContainerStarted","Data":"fc6286c4dd9535f2dc44727fc082d8ed46494d124855beade643ddf6bd66c786"} Nov 02 21:28:01.893689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:01.893650 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:05.572766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:05.572731 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:28:05.591196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:05.591174 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:28:06.055084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:06.054994 4441 generic.go:356] "Generic (PLEG): container finished" podID="d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" containerID="eb649dea75c7d2ec8a4c7a5b3c1390262a499f981bbacc6a40cf03ac09857537" exitCode=0 Nov 02 21:28:06.055084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:06.055067 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" event={"ID":"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda","Type":"ContainerDied","Data":"eb649dea75c7d2ec8a4c7a5b3c1390262a499f981bbacc6a40cf03ac09857537"} Nov 02 21:28:06.071831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:06.071804 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:28:06.420735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:06.420702 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:06.536576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:06.536546 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:07.218448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.218411 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:28:07.249104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.249076 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-openshift-ca\") pod \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " Nov 02 21:28:07.249235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.249117 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-kube-api-access-bn5qb\" (UniqueName: \"kubernetes.io/projected/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-kube-api-access-bn5qb\") pod \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " Nov 02 21:28:07.249235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.249194 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-xks-ca\") pod \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\" (UID: \"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda\") " Nov 02 21:28:07.249456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.249432 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-openshift-ca" pod "d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" (UID: "d69c6c9d-23d3-4be2-ad09-158a6cf8dbda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:07.249583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.249564 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:07.251917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.251890 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-xks-ca" pod "d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" (UID: "d69c6c9d-23d3-4be2-ad09-158a6cf8dbda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:07.251917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.251907 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-kube-api-access-bn5qb" pod "d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" (UID: "d69c6c9d-23d3-4be2-ad09-158a6cf8dbda"). InnerVolumeSpecName "kube-api-access-bn5qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:07.350461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.350391 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:07.350461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:07.350419 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn5qb\" (UniqueName: \"kubernetes.io/projected/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda-kube-api-access-bn5qb\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:08.070675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:08.070644 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" Nov 02 21:28:08.070839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:08.070643 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc" event={"ID":"d69c6c9d-23d3-4be2-ad09-158a6cf8dbda","Type":"ContainerDied","Data":"fc6286c4dd9535f2dc44727fc082d8ed46494d124855beade643ddf6bd66c786"} Nov 02 21:28:08.070839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:08.070772 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6286c4dd9535f2dc44727fc082d8ed46494d124855beade643ddf6bd66c786" Nov 02 21:28:10.613833 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:28:10.613795 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:28:10.973129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:10.973036 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:11.351429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:11.351393 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:15.481126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:15.481091 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:19.967297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:19.967260 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:20.531574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.531536 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn"] Nov 02 21:28:20.532264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.532238 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" containerName="revoke-keys" Nov 02 21:28:20.532264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.532261 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" containerName="revoke-keys" Nov 02 21:28:20.532465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.532387 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" containerName="revoke-keys" Nov 02 21:28:20.536080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.536059 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.588218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.588187 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn"] Nov 02 21:28:20.602337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.602308 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwlxx\" (UniqueName: \"kubernetes.io/projected/38b5cee1-49ba-4747-bf53-be0892a2a431-kube-api-access-jwlxx\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.602498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.602359 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.602742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.602719 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.630626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.630597 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:28:20.630626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.630602 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:28:20.630811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.630654 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:28:20.632566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.632548 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:28:20.703986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.703943 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.704170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.704066 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwlxx\" (UniqueName: \"kubernetes.io/projected/38b5cee1-49ba-4747-bf53-be0892a2a431-kube-api-access-jwlxx\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.704170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.704099 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.704759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.704738 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.706382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.706356 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.758096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.758060 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwlxx\" (UniqueName: \"kubernetes.io/projected/38b5cee1-49ba-4747-bf53-be0892a2a431-kube-api-access-jwlxx\") pod \"maas-api-revoke-sub-49e6b02b-5lgrn\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:20.851811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:20.851773 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:21.059236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.059196 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f"] Nov 02 21:28:21.063582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.063558 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.108469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.108404 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.108469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.108440 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccw4z\" (UniqueName: \"kubernetes.io/projected/3c2f4ac1-039c-4ce3-be32-abb3af240960-kube-api-access-ccw4z\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.108650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.108583 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.116054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.116022 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f"] Nov 02 21:28:21.209958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.209923 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.210122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.209988 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccw4z\" (UniqueName: \"kubernetes.io/projected/3c2f4ac1-039c-4ce3-be32-abb3af240960-kube-api-access-ccw4z\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.210122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.210084 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.210778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.210749 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.212627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.212600 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.264732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.264698 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccw4z\" (UniqueName: \"kubernetes.io/projected/3c2f4ac1-039c-4ce3-be32-abb3af240960-kube-api-access-ccw4z\") pod \"maas-api-revoke-sub-376d2436-qmr2f\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.287693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.287666 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn"] Nov 02 21:28:21.293187 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:28:21.293161 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b5cee1_49ba_4747_bf53_be0892a2a431.slice/crio-66ddbbea4aeedfd655022f294e80ee1489ac2636e1a1f5e779874422ba6eb65d WatchSource:0}: Error finding container 66ddbbea4aeedfd655022f294e80ee1489ac2636e1a1f5e779874422ba6eb65d: Status 404 returned error can't find the container with id 66ddbbea4aeedfd655022f294e80ee1489ac2636e1a1f5e779874422ba6eb65d Nov 02 21:28:21.378729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.378671 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:21.759433 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:21.759357 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f"] Nov 02 21:28:21.765002 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:28:21.764950 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2f4ac1_039c_4ce3_be32_abb3af240960.slice/crio-d0918404d4ea3f9f57b4bc39ed1b48b03a6cb065104201a0d8d51fdcb59ac6c6 WatchSource:0}: Error finding container d0918404d4ea3f9f57b4bc39ed1b48b03a6cb065104201a0d8d51fdcb59ac6c6: Status 404 returned error can't find the container with id d0918404d4ea3f9f57b4bc39ed1b48b03a6cb065104201a0d8d51fdcb59ac6c6 Nov 02 21:28:22.177162 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:22.177120 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" event={"ID":"3c2f4ac1-039c-4ce3-be32-abb3af240960","Type":"ContainerStarted","Data":"fabca8f225c236d32ad601003736bf72562df70c6944333c0d29b38a83dff932"} Nov 02 21:28:22.177570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:22.177217 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" event={"ID":"3c2f4ac1-039c-4ce3-be32-abb3af240960","Type":"ContainerStarted","Data":"d0918404d4ea3f9f57b4bc39ed1b48b03a6cb065104201a0d8d51fdcb59ac6c6"} Nov 02 21:28:22.179246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:22.179213 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" event={"ID":"38b5cee1-49ba-4747-bf53-be0892a2a431","Type":"ContainerStarted","Data":"38043e4c8631c6d334d2ac146c1262383e8f97875f94acda63f494d14b2dd187"} Nov 02 21:28:22.179415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:22.179266 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" event={"ID":"38b5cee1-49ba-4747-bf53-be0892a2a431","Type":"ContainerStarted","Data":"66ddbbea4aeedfd655022f294e80ee1489ac2636e1a1f5e779874422ba6eb65d"} Nov 02 21:28:22.287177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:22.287128 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" podStartSLOduration=1.287114656 podStartE2EDuration="1.287114656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:22.283466017 +0000 UTC m=+931.839094355" watchObservedRunningTime="2026-11-02 21:28:22.287114656 +0000 UTC m=+931.842742993" Nov 02 21:28:22.393303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:22.393209 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" podStartSLOduration=2.393193803 podStartE2EDuration="2.393193803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:22.389704948 +0000 UTC m=+931.945333286" watchObservedRunningTime="2026-11-02 21:28:22.393193803 +0000 UTC m=+931.948822186" Nov 02 21:28:23.188007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:23.187946 4441 generic.go:356] "Generic (PLEG): container finished" podID="38b5cee1-49ba-4747-bf53-be0892a2a431" containerID="38043e4c8631c6d334d2ac146c1262383e8f97875f94acda63f494d14b2dd187" exitCode=0 Nov 02 21:28:23.188470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:23.188036 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" event={"ID":"38b5cee1-49ba-4747-bf53-be0892a2a431","Type":"ContainerDied","Data":"38043e4c8631c6d334d2ac146c1262383e8f97875f94acda63f494d14b2dd187"} Nov 02 21:28:23.189983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:23.189929 4441 generic.go:356] "Generic (PLEG): container finished" podID="3c2f4ac1-039c-4ce3-be32-abb3af240960" containerID="fabca8f225c236d32ad601003736bf72562df70c6944333c0d29b38a83dff932" exitCode=0 Nov 02 21:28:23.190103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:23.189958 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" event={"ID":"3c2f4ac1-039c-4ce3-be32-abb3af240960","Type":"ContainerDied","Data":"fabca8f225c236d32ad601003736bf72562df70c6944333c0d29b38a83dff932"} Nov 02 21:28:24.358253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.358230 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:24.396774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.396740 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:24.443916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.443888 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-openshift-ca\") pod \"38b5cee1-49ba-4747-bf53-be0892a2a431\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " Nov 02 21:28:24.444100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.443935 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b5cee1-49ba-4747-bf53-be0892a2a431-kube-api-access-jwlxx\" (UniqueName: \"kubernetes.io/projected/38b5cee1-49ba-4747-bf53-be0892a2a431-kube-api-access-jwlxx\") pod \"38b5cee1-49ba-4747-bf53-be0892a2a431\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " Nov 02 21:28:24.444100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.444069 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-xks-ca\") pod \"38b5cee1-49ba-4747-bf53-be0892a2a431\" (UID: \"38b5cee1-49ba-4747-bf53-be0892a2a431\") " Nov 02 21:28:24.444295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.444272 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-openshift-ca" pod "38b5cee1-49ba-4747-bf53-be0892a2a431" (UID: "38b5cee1-49ba-4747-bf53-be0892a2a431"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:24.444469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.444454 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:24.446158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.446134 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-xks-ca" pod "38b5cee1-49ba-4747-bf53-be0892a2a431" (UID: "38b5cee1-49ba-4747-bf53-be0892a2a431"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:24.446264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.446167 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b5cee1-49ba-4747-bf53-be0892a2a431-kube-api-access-jwlxx" pod "38b5cee1-49ba-4747-bf53-be0892a2a431" (UID: "38b5cee1-49ba-4747-bf53-be0892a2a431"). InnerVolumeSpecName "kube-api-access-jwlxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:24.545227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.545197 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2f4ac1-039c-4ce3-be32-abb3af240960-kube-api-access-ccw4z\" (UniqueName: \"kubernetes.io/projected/3c2f4ac1-039c-4ce3-be32-abb3af240960-kube-api-access-ccw4z\") pod \"3c2f4ac1-039c-4ce3-be32-abb3af240960\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " Nov 02 21:28:24.545391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.545285 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-openshift-ca\") pod \"3c2f4ac1-039c-4ce3-be32-abb3af240960\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " Nov 02 21:28:24.545473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.545455 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-xks-ca\") pod \"3c2f4ac1-039c-4ce3-be32-abb3af240960\" (UID: \"3c2f4ac1-039c-4ce3-be32-abb3af240960\") " Nov 02 21:28:24.545690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.545664 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-openshift-ca" pod "3c2f4ac1-039c-4ce3-be32-abb3af240960" (UID: "3c2f4ac1-039c-4ce3-be32-abb3af240960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:24.545841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.545827 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwlxx\" (UniqueName: \"kubernetes.io/projected/38b5cee1-49ba-4747-bf53-be0892a2a431-kube-api-access-jwlxx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:24.545892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.545841 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b5cee1-49ba-4747-bf53-be0892a2a431-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:24.545892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.545851 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:24.547192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.547173 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2f4ac1-039c-4ce3-be32-abb3af240960-kube-api-access-ccw4z" pod "3c2f4ac1-039c-4ce3-be32-abb3af240960" (UID: "3c2f4ac1-039c-4ce3-be32-abb3af240960"). InnerVolumeSpecName "kube-api-access-ccw4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:24.547325 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.547305 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-xks-ca" pod "3c2f4ac1-039c-4ce3-be32-abb3af240960" (UID: "3c2f4ac1-039c-4ce3-be32-abb3af240960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:24.580058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.579934 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:24.647213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.647178 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c2f4ac1-039c-4ce3-be32-abb3af240960-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:24.647398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:24.647224 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccw4z\" (UniqueName: \"kubernetes.io/projected/3c2f4ac1-039c-4ce3-be32-abb3af240960-kube-api-access-ccw4z\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:25.207448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:25.207416 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" Nov 02 21:28:25.207448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:25.207426 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f" event={"ID":"3c2f4ac1-039c-4ce3-be32-abb3af240960","Type":"ContainerDied","Data":"d0918404d4ea3f9f57b4bc39ed1b48b03a6cb065104201a0d8d51fdcb59ac6c6"} Nov 02 21:28:25.207716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:25.207483 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0918404d4ea3f9f57b4bc39ed1b48b03a6cb065104201a0d8d51fdcb59ac6c6" Nov 02 21:28:25.209104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:25.209080 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" Nov 02 21:28:25.209104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:25.209089 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn" event={"ID":"38b5cee1-49ba-4747-bf53-be0892a2a431","Type":"ContainerDied","Data":"66ddbbea4aeedfd655022f294e80ee1489ac2636e1a1f5e779874422ba6eb65d"} Nov 02 21:28:25.209259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:25.209136 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ddbbea4aeedfd655022f294e80ee1489ac2636e1a1f5e779874422ba6eb65d" Nov 02 21:28:29.123086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:29.123050 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:33.595450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:33.595345 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:36.088861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.088824 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5"] Nov 02 21:28:36.089342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.089325 4441 cpu_manager.go:414] "containerMap: removing container" podUID="38b5cee1-49ba-4747-bf53-be0892a2a431" containerName="revoke-keys" Nov 02 21:28:36.089390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.089342 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b5cee1-49ba-4747-bf53-be0892a2a431" containerName="revoke-keys" Nov 02 21:28:36.089390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.089372 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3c2f4ac1-039c-4ce3-be32-abb3af240960" containerName="revoke-keys" Nov 02 21:28:36.089390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.089378 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2f4ac1-039c-4ce3-be32-abb3af240960" containerName="revoke-keys" Nov 02 21:28:36.089483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.089455 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2f4ac1-039c-4ce3-be32-abb3af240960" containerName="revoke-keys" Nov 02 21:28:36.089483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.089466 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b5cee1-49ba-4747-bf53-be0892a2a431" containerName="revoke-keys" Nov 02 21:28:36.098049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.098027 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.146542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.146514 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5"] Nov 02 21:28:36.188666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.188637 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:28:36.188666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.188662 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:28:36.194356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.194337 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:28:36.194356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.194349 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:28:36.266494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.266465 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.266657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.266543 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.266657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.266582 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qph2w\" (UniqueName: \"kubernetes.io/projected/49dfb678-11b2-4620-832b-7e1db1b871ab-kube-api-access-qph2w\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.367487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.367413 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.367657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.367483 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qph2w\" (UniqueName: \"kubernetes.io/projected/49dfb678-11b2-4620-832b-7e1db1b871ab-kube-api-access-qph2w\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.367657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.367580 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.368261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.368240 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.369836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.369817 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.421006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.420962 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qph2w\" (UniqueName: \"kubernetes.io/projected/49dfb678-11b2-4620-832b-7e1db1b871ab-kube-api-access-qph2w\") pod \"maas-api-revoke-sub-98a408e0-757j5\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:36.711809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:36.711742 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:37.085341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:37.085306 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5"] Nov 02 21:28:37.091504 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:28:37.091478 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49dfb678_11b2_4620_832b_7e1db1b871ab.slice/crio-d0510951c289559b0571e3340eddc6f3016b8ae908fa2640a9b068563a349929 WatchSource:0}: Error finding container d0510951c289559b0571e3340eddc6f3016b8ae908fa2640a9b068563a349929: Status 404 returned error can't find the container with id d0510951c289559b0571e3340eddc6f3016b8ae908fa2640a9b068563a349929 Nov 02 21:28:37.294073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:37.294043 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" event={"ID":"49dfb678-11b2-4620-832b-7e1db1b871ab","Type":"ContainerStarted","Data":"d0510951c289559b0571e3340eddc6f3016b8ae908fa2640a9b068563a349929"} Nov 02 21:28:38.085371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:38.085337 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:38.303600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:38.303498 4441 generic.go:356] "Generic (PLEG): container finished" podID="49dfb678-11b2-4620-832b-7e1db1b871ab" containerID="bade7190ef34ee275a4cf6f7424d1763e183ece0c4e326cedca7bbd851a4880d" exitCode=0 Nov 02 21:28:38.304110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:38.303591 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" event={"ID":"49dfb678-11b2-4620-832b-7e1db1b871ab","Type":"ContainerDied","Data":"bade7190ef34ee275a4cf6f7424d1763e183ece0c4e326cedca7bbd851a4880d"} Nov 02 21:28:39.109385 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.109353 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:39.228061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.228031 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:39.463894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.463872 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:39.605150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.605122 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-openshift-ca\") pod \"49dfb678-11b2-4620-832b-7e1db1b871ab\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " Nov 02 21:28:39.605320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.605169 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49dfb678-11b2-4620-832b-7e1db1b871ab-kube-api-access-qph2w\" (UniqueName: \"kubernetes.io/projected/49dfb678-11b2-4620-832b-7e1db1b871ab-kube-api-access-qph2w\") pod \"49dfb678-11b2-4620-832b-7e1db1b871ab\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " Nov 02 21:28:39.605320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.605217 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-xks-ca\") pod \"49dfb678-11b2-4620-832b-7e1db1b871ab\" (UID: \"49dfb678-11b2-4620-832b-7e1db1b871ab\") " Nov 02 21:28:39.605514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.605490 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-openshift-ca" pod "49dfb678-11b2-4620-832b-7e1db1b871ab" (UID: "49dfb678-11b2-4620-832b-7e1db1b871ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:39.607365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.607341 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-xks-ca" pod "49dfb678-11b2-4620-832b-7e1db1b871ab" (UID: "49dfb678-11b2-4620-832b-7e1db1b871ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:39.607365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.607347 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49dfb678-11b2-4620-832b-7e1db1b871ab-kube-api-access-qph2w" pod "49dfb678-11b2-4620-832b-7e1db1b871ab" (UID: "49dfb678-11b2-4620-832b-7e1db1b871ab"). InnerVolumeSpecName "kube-api-access-qph2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:39.706826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.706741 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:39.706826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.706772 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qph2w\" (UniqueName: \"kubernetes.io/projected/49dfb678-11b2-4620-832b-7e1db1b871ab-kube-api-access-qph2w\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:39.706826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:39.706789 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49dfb678-11b2-4620-832b-7e1db1b871ab-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:40.323399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:40.323373 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" Nov 02 21:28:40.323542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:40.323367 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5" event={"ID":"49dfb678-11b2-4620-832b-7e1db1b871ab","Type":"ContainerDied","Data":"d0510951c289559b0571e3340eddc6f3016b8ae908fa2640a9b068563a349929"} Nov 02 21:28:40.323542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:40.323525 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0510951c289559b0571e3340eddc6f3016b8ae908fa2640a9b068563a349929" Nov 02 21:28:40.624687 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:28:40.624604 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:28:41.364459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.364425 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv"] Nov 02 21:28:41.365003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.364982 4441 cpu_manager.go:414] "containerMap: removing container" podUID="49dfb678-11b2-4620-832b-7e1db1b871ab" containerName="revoke-keys" Nov 02 21:28:41.365003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.365003 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49dfb678-11b2-4620-832b-7e1db1b871ab" containerName="revoke-keys" Nov 02 21:28:41.365180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.365172 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="49dfb678-11b2-4620-832b-7e1db1b871ab" containerName="revoke-keys" Nov 02 21:28:41.368802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.368780 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.422487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.422448 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv"] Nov 02 21:28:41.464984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.464934 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:28:41.464984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.464955 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:28:41.464984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.464934 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:28:41.465199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.464934 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:28:41.523388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.523349 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.523567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.523397 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjq5\" (UniqueName: \"kubernetes.io/projected/0d01c62f-bba3-476e-98c3-603e69e6ecf3-kube-api-access-4mjq5\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.523567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.523556 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.624684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.624586 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.624684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.624628 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjq5\" (UniqueName: \"kubernetes.io/projected/0d01c62f-bba3-476e-98c3-603e69e6ecf3-kube-api-access-4mjq5\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.624684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.624686 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.625383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.625361 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.627720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.627695 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.681497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.681467 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjq5\" (UniqueName: \"kubernetes.io/projected/0d01c62f-bba3-476e-98c3-603e69e6ecf3-kube-api-access-4mjq5\") pod \"maas-api-revoke-sub-4dff1ca0-5kxqv\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:41.687264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:41.687239 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:42.072538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:42.072510 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv"] Nov 02 21:28:42.075015 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:28:42.074584 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d01c62f_bba3_476e_98c3_603e69e6ecf3.slice/crio-b35b4bc571b91f27d0d18a84acb2dbbeeafd6c69e90eef6d61308b08d254cc48 WatchSource:0}: Error finding container b35b4bc571b91f27d0d18a84acb2dbbeeafd6c69e90eef6d61308b08d254cc48: Status 404 returned error can't find the container with id b35b4bc571b91f27d0d18a84acb2dbbeeafd6c69e90eef6d61308b08d254cc48 Nov 02 21:28:42.338376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:42.338302 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" event={"ID":"0d01c62f-bba3-476e-98c3-603e69e6ecf3","Type":"ContainerStarted","Data":"b35b4bc571b91f27d0d18a84acb2dbbeeafd6c69e90eef6d61308b08d254cc48"} Nov 02 21:28:42.696152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:42.696058 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:43.346385 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:43.346347 4441 generic.go:356] "Generic (PLEG): container finished" podID="0d01c62f-bba3-476e-98c3-603e69e6ecf3" containerID="c1233461bc1e90032503d93a2bc1674a1d73eff360a0638851467e547bfa7271" exitCode=0 Nov 02 21:28:43.346621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:43.346389 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" event={"ID":"0d01c62f-bba3-476e-98c3-603e69e6ecf3","Type":"ContainerDied","Data":"c1233461bc1e90032503d93a2bc1674a1d73eff360a0638851467e547bfa7271"} Nov 02 21:28:44.741492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.741455 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:44.889483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.889402 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d01c62f-bba3-476e-98c3-603e69e6ecf3-kube-api-access-4mjq5\" (UniqueName: \"kubernetes.io/projected/0d01c62f-bba3-476e-98c3-603e69e6ecf3-kube-api-access-4mjq5\") pod \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " Nov 02 21:28:44.889637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.889561 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-xks-ca\") pod \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " Nov 02 21:28:44.889637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.889627 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-openshift-ca\") pod \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\" (UID: \"0d01c62f-bba3-476e-98c3-603e69e6ecf3\") " Nov 02 21:28:44.890209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.890179 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-openshift-ca" pod "0d01c62f-bba3-476e-98c3-603e69e6ecf3" (UID: "0d01c62f-bba3-476e-98c3-603e69e6ecf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:44.892167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.892141 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-xks-ca" pod "0d01c62f-bba3-476e-98c3-603e69e6ecf3" (UID: "0d01c62f-bba3-476e-98c3-603e69e6ecf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:44.892260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.892161 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d01c62f-bba3-476e-98c3-603e69e6ecf3-kube-api-access-4mjq5" pod "0d01c62f-bba3-476e-98c3-603e69e6ecf3" (UID: "0d01c62f-bba3-476e-98c3-603e69e6ecf3"). InnerVolumeSpecName "kube-api-access-4mjq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:44.991077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.991045 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:44.991077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.991073 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjq5\" (UniqueName: \"kubernetes.io/projected/0d01c62f-bba3-476e-98c3-603e69e6ecf3-kube-api-access-4mjq5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:44.991077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:44.991083 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d01c62f-bba3-476e-98c3-603e69e6ecf3-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:45.376063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:45.376022 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" Nov 02 21:28:45.376063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:45.376043 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv" event={"ID":"0d01c62f-bba3-476e-98c3-603e69e6ecf3","Type":"ContainerDied","Data":"b35b4bc571b91f27d0d18a84acb2dbbeeafd6c69e90eef6d61308b08d254cc48"} Nov 02 21:28:45.376268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:45.376101 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b35b4bc571b91f27d0d18a84acb2dbbeeafd6c69e90eef6d61308b08d254cc48" Nov 02 21:28:45.783300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:45.783206 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:47.194170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:47.194131 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:51.688051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:51.688008 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:52.118275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.118241 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6"] Nov 02 21:28:52.118719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.118707 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0d01c62f-bba3-476e-98c3-603e69e6ecf3" containerName="revoke-keys" Nov 02 21:28:52.118763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.118721 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d01c62f-bba3-476e-98c3-603e69e6ecf3" containerName="revoke-keys" Nov 02 21:28:52.118836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.118824 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d01c62f-bba3-476e-98c3-603e69e6ecf3" containerName="revoke-keys" Nov 02 21:28:52.122415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.122397 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.174687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.174659 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6"] Nov 02 21:28:52.217432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.217405 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:28:52.217432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.217405 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:28:52.217660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.217642 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:28:52.217718 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.217645 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:28:52.269475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.269446 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.269602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.269488 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.269654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.269628 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85zb\" (UniqueName: \"kubernetes.io/projected/8e84eff7-4414-406d-9026-f029e62d977a-kube-api-access-x85zb\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.370875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.370810 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.370875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.370855 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.371071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.370903 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85zb\" (UniqueName: \"kubernetes.io/projected/8e84eff7-4414-406d-9026-f029e62d977a-kube-api-access-x85zb\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.371612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.371591 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.373159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.373136 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.424861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.424830 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85zb\" (UniqueName: \"kubernetes.io/projected/8e84eff7-4414-406d-9026-f029e62d977a-kube-api-access-x85zb\") pod \"maas-api-revoke-sub-8963e511-f5cg6\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.435436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.435414 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:52.806845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:52.806817 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6"] Nov 02 21:28:52.812082 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:28:52.812052 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e84eff7_4414_406d_9026_f029e62d977a.slice/crio-8c16f0aade672962599d726fbcda87f37c1f1dcb5ad6f0961d3f9b49d3b471e9 WatchSource:0}: Error finding container 8c16f0aade672962599d726fbcda87f37c1f1dcb5ad6f0961d3f9b49d3b471e9: Status 404 returned error can't find the container with id 8c16f0aade672962599d726fbcda87f37c1f1dcb5ad6f0961d3f9b49d3b471e9 Nov 02 21:28:53.439450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:53.439352 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" event={"ID":"8e84eff7-4414-406d-9026-f029e62d977a","Type":"ContainerStarted","Data":"ba520c7609ce434daad497d0a77af5b01a641042120fc44a444a0520400ef15f"} Nov 02 21:28:53.439450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:53.439406 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" event={"ID":"8e84eff7-4414-406d-9026-f029e62d977a","Type":"ContainerStarted","Data":"8c16f0aade672962599d726fbcda87f37c1f1dcb5ad6f0961d3f9b49d3b471e9"} Nov 02 21:28:53.549658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:53.549606 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" podStartSLOduration=1.549592397 podStartE2EDuration="1.549592397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:53.546259378 +0000 UTC m=+963.101887694" watchObservedRunningTime="2026-11-02 21:28:53.549592397 +0000 UTC m=+963.105220733" Nov 02 21:28:54.455696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:54.455658 4441 generic.go:356] "Generic (PLEG): container finished" podID="8e84eff7-4414-406d-9026-f029e62d977a" containerID="ba520c7609ce434daad497d0a77af5b01a641042120fc44a444a0520400ef15f" exitCode=0 Nov 02 21:28:54.456158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:54.455727 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" event={"ID":"8e84eff7-4414-406d-9026-f029e62d977a","Type":"ContainerDied","Data":"ba520c7609ce434daad497d0a77af5b01a641042120fc44a444a0520400ef15f"} Nov 02 21:28:55.402889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.402855 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw"] Nov 02 21:28:55.403126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.403103 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" containerID="cri-o://c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228" gracePeriod=30 Nov 02 21:28:55.615617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.615587 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:55.712881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.712808 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e84eff7-4414-406d-9026-f029e62d977a-kube-api-access-x85zb\" (UniqueName: \"kubernetes.io/projected/8e84eff7-4414-406d-9026-f029e62d977a-kube-api-access-x85zb\") pod \"8e84eff7-4414-406d-9026-f029e62d977a\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " Nov 02 21:28:55.713045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.712914 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-xks-ca\") pod \"8e84eff7-4414-406d-9026-f029e62d977a\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " Nov 02 21:28:55.713125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.713108 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-openshift-ca\") pod \"8e84eff7-4414-406d-9026-f029e62d977a\" (UID: \"8e84eff7-4414-406d-9026-f029e62d977a\") " Nov 02 21:28:55.713468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.713430 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-openshift-ca" pod "8e84eff7-4414-406d-9026-f029e62d977a" (UID: "8e84eff7-4414-406d-9026-f029e62d977a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:55.713600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.713575 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:55.715029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.715002 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e84eff7-4414-406d-9026-f029e62d977a-kube-api-access-x85zb" pod "8e84eff7-4414-406d-9026-f029e62d977a" (UID: "8e84eff7-4414-406d-9026-f029e62d977a"). InnerVolumeSpecName "kube-api-access-x85zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:55.715029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.715010 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-xks-ca" pod "8e84eff7-4414-406d-9026-f029e62d977a" (UID: "8e84eff7-4414-406d-9026-f029e62d977a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:55.814760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.814731 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x85zb\" (UniqueName: \"kubernetes.io/projected/8e84eff7-4414-406d-9026-f029e62d977a-kube-api-access-x85zb\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:55.814760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:55.814758 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e84eff7-4414-406d-9026-f029e62d977a-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:56.157018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:56.156983 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:56.473491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:56.473421 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" Nov 02 21:28:56.473491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:56.473463 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6" event={"ID":"8e84eff7-4414-406d-9026-f029e62d977a","Type":"ContainerDied","Data":"8c16f0aade672962599d726fbcda87f37c1f1dcb5ad6f0961d3f9b49d3b471e9"} Nov 02 21:28:56.473661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:28:56.473511 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c16f0aade672962599d726fbcda87f37c1f1dcb5ad6f0961d3f9b49d3b471e9" Nov 02 21:29:00.664368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.664340 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:29:00.694661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.694600 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:00.770349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770319 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-envoy\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770365 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-podinfo\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770395 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-credential-socket\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770425 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-kube-api-access-m58bz\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-kube-api-access-m58bz\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770485 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-token\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-token\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770534 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-socket\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770604 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-certs\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770674 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istiod-ca-cert\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770732 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-data\") pod \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\" (UID: \"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190\") " Nov 02 21:29:00.770810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770735 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-credential-socket" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:29:00.771062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770898 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-socket" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:29:00.771062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.770917 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-certs" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:29:00.771180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.771117 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-data" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:29:00.771180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.771157 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istiod-ca-cert" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:00.771307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.771283 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.771368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.771311 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.771368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.771326 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.771368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.771339 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.771522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.771352 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.772934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.772906 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-podinfo" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:29:00.773071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.772930 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-token" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:00.773071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.773024 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-envoy" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:29:00.773071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.773021 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-kube-api-access-m58bz" pod "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" (UID: "0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190"). InnerVolumeSpecName "kube-api-access-m58bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:00.872009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.871954 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.872009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.872009 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.872189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.872021 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m58bz\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-kube-api-access-m58bz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:00.872189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:00.872032 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:01.510617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.510585 4441 generic.go:356] "Generic (PLEG): container finished" podID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerID="c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228" exitCode=0 Nov 02 21:29:01.510795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.510644 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" Nov 02 21:29:01.510795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.510672 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" event={"ID":"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190","Type":"ContainerDied","Data":"c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228"} Nov 02 21:29:01.510795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.510715 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw" event={"ID":"0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190","Type":"ContainerDied","Data":"c2c5ab65cc32db13a4e863a6988849b65bc45c330e6d6d6a588663c217c79135"} Nov 02 21:29:01.510795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.510736 4441 scope.go:122] "RemoveContainer" containerID="c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228" Nov 02 21:29:01.522310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.522289 4441 scope.go:122] "RemoveContainer" containerID="47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b" Nov 02 21:29:01.533831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.533692 4441 scope.go:122] "RemoveContainer" containerID="c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228" Nov 02 21:29:01.533962 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:29:01.533944 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228\": container with ID starting with c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228 not found: ID does not exist" containerID="c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228" Nov 02 21:29:01.534025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.533984 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228"} err="failed to get container status \"c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228\": rpc error: code = NotFound desc = could not find container \"c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228\": container with ID starting with c24b8eca78f4b801936a60f98e9762383f579875275a4f8a201a857702fcb228 not found: ID does not exist" Nov 02 21:29:01.534025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.534003 4441 scope.go:122] "RemoveContainer" containerID="47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b" Nov 02 21:29:01.534237 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:29:01.534219 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b\": container with ID starting with 47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b not found: ID does not exist" containerID="47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b" Nov 02 21:29:01.534276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.534243 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b"} err="failed to get container status \"47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b\": rpc error: code = NotFound desc = could not find container \"47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b\": container with ID starting with 47c44383a2c30f3e571eb71ba7063da49f649f4f50c0b36befe4fbab451e550b not found: ID does not exist" Nov 02 21:29:01.719257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.719222 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw"] Nov 02 21:29:01.725499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:01.725469 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wkbnw"] Nov 02 21:29:02.583485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:02.583446 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" path="/var/lib/kubelet/pods/0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190/volumes" Nov 02 21:29:05.178526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.178484 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:05.546104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546035 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4"] Nov 02 21:29:05.546519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546505 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" Nov 02 21:29:05.546577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546521 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" Nov 02 21:29:05.546577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546561 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8e84eff7-4414-406d-9026-f029e62d977a" containerName="revoke-keys" Nov 02 21:29:05.546577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546566 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e84eff7-4414-406d-9026-f029e62d977a" containerName="revoke-keys" Nov 02 21:29:05.546672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546641 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" Nov 02 21:29:05.546672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546650 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" Nov 02 21:29:05.546672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.546662 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e84eff7-4414-406d-9026-f029e62d977a" containerName="revoke-keys" Nov 02 21:29:05.551706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.551687 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.603602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.603563 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4"] Nov 02 21:29:05.621549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.621521 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6fk7\" (UniqueName: \"kubernetes.io/projected/9017c47f-0410-44ca-a546-f4be35713947-kube-api-access-w6fk7\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.621729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.621650 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9017c47f-0410-44ca-a546-f4be35713947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.621729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.621699 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9017c47f-0410-44ca-a546-f4be35713947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.645596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.645571 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:29:05.645730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.645632 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:29:05.645730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.645688 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:29:05.648338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.648311 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:29:05.723039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.723009 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9017c47f-0410-44ca-a546-f4be35713947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.723195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.723063 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9017c47f-0410-44ca-a546-f4be35713947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.723195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.723146 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6fk7\" (UniqueName: \"kubernetes.io/projected/9017c47f-0410-44ca-a546-f4be35713947-kube-api-access-w6fk7\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.723792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.723769 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9017c47f-0410-44ca-a546-f4be35713947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.725310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.725289 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9017c47f-0410-44ca-a546-f4be35713947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.776831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.776808 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6fk7\" (UniqueName: \"kubernetes.io/projected/9017c47f-0410-44ca-a546-f4be35713947-kube-api-access-w6fk7\") pod \"maas-api-revoke-sub-2fde3573-dbwr4\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:05.864859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:05.864836 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:06.244046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:06.243952 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4"] Nov 02 21:29:06.249697 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:06.249666 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9017c47f_0410_44ca_a546_f4be35713947.slice/crio-6d4d948381cf6fcb606ccf792670cb5b1579e780ae5071c99b60d76a68b3fbac WatchSource:0}: Error finding container 6d4d948381cf6fcb606ccf792670cb5b1579e780ae5071c99b60d76a68b3fbac: Status 404 returned error can't find the container with id 6d4d948381cf6fcb606ccf792670cb5b1579e780ae5071c99b60d76a68b3fbac Nov 02 21:29:06.546675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:06.546605 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" event={"ID":"9017c47f-0410-44ca-a546-f4be35713947","Type":"ContainerStarted","Data":"6d4d948381cf6fcb606ccf792670cb5b1579e780ae5071c99b60d76a68b3fbac"} Nov 02 21:29:07.567625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:07.567591 4441 generic.go:356] "Generic (PLEG): container finished" podID="9017c47f-0410-44ca-a546-f4be35713947" containerID="31ccb6d13c1119a96a119e3fa8eec882fff128cd1921e689818c274793a9dce5" exitCode=0 Nov 02 21:29:07.568033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:07.567664 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" event={"ID":"9017c47f-0410-44ca-a546-f4be35713947","Type":"ContainerDied","Data":"31ccb6d13c1119a96a119e3fa8eec882fff128cd1921e689818c274793a9dce5"} Nov 02 21:29:08.723885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.723849 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:08.882570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.882482 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9017c47f-0410-44ca-a546-f4be35713947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9017c47f-0410-44ca-a546-f4be35713947-maas-api-openshift-ca\") pod \"9017c47f-0410-44ca-a546-f4be35713947\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " Nov 02 21:29:08.882716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.882673 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9017c47f-0410-44ca-a546-f4be35713947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9017c47f-0410-44ca-a546-f4be35713947-maas-api-xks-ca\") pod \"9017c47f-0410-44ca-a546-f4be35713947\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " Nov 02 21:29:08.882857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.882832 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9017c47f-0410-44ca-a546-f4be35713947-kube-api-access-w6fk7\" (UniqueName: \"kubernetes.io/projected/9017c47f-0410-44ca-a546-f4be35713947-kube-api-access-w6fk7\") pod \"9017c47f-0410-44ca-a546-f4be35713947\" (UID: \"9017c47f-0410-44ca-a546-f4be35713947\") " Nov 02 21:29:08.882857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.882847 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9017c47f-0410-44ca-a546-f4be35713947-maas-api-openshift-ca" pod "9017c47f-0410-44ca-a546-f4be35713947" (UID: "9017c47f-0410-44ca-a546-f4be35713947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:08.883216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.883188 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9017c47f-0410-44ca-a546-f4be35713947-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:08.884865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.884835 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9017c47f-0410-44ca-a546-f4be35713947-maas-api-xks-ca" pod "9017c47f-0410-44ca-a546-f4be35713947" (UID: "9017c47f-0410-44ca-a546-f4be35713947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:08.885111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.885089 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9017c47f-0410-44ca-a546-f4be35713947-kube-api-access-w6fk7" pod "9017c47f-0410-44ca-a546-f4be35713947" (UID: "9017c47f-0410-44ca-a546-f4be35713947"). InnerVolumeSpecName "kube-api-access-w6fk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:08.984298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.984254 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6fk7\" (UniqueName: \"kubernetes.io/projected/9017c47f-0410-44ca-a546-f4be35713947-kube-api-access-w6fk7\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:08.984298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:08.984300 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9017c47f-0410-44ca-a546-f4be35713947-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:09.357079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.357041 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5"] Nov 02 21:29:09.357771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.357753 4441 cpu_manager.go:414] "containerMap: removing container" podUID="9017c47f-0410-44ca-a546-f4be35713947" containerName="revoke-keys" Nov 02 21:29:09.357816 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.357774 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9017c47f-0410-44ca-a546-f4be35713947" containerName="revoke-keys" Nov 02 21:29:09.357852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.357820 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" Nov 02 21:29:09.357852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.357828 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf8cbb8-f7f8-4fd4-882c-3b9cc4307190" containerName="istio-proxy" Nov 02 21:29:09.357954 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.357943 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="9017c47f-0410-44ca-a546-f4be35713947" containerName="revoke-keys" Nov 02 21:29:09.360285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.360267 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.415176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.415140 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5"] Nov 02 21:29:09.492991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.489890 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.492991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.490029 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2sl\" (UniqueName: \"kubernetes.io/projected/240c5a07-0957-4f3e-94df-d35d7dfc420b-kube-api-access-xk2sl\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.492991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.490110 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.590829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.590790 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.591314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.591291 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.591531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.591516 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2sl\" (UniqueName: \"kubernetes.io/projected/240c5a07-0957-4f3e-94df-d35d7dfc420b-kube-api-access-xk2sl\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.598992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.597535 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.599779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.599755 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.638003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.637905 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" event={"ID":"9017c47f-0410-44ca-a546-f4be35713947","Type":"ContainerDied","Data":"6d4d948381cf6fcb606ccf792670cb5b1579e780ae5071c99b60d76a68b3fbac"} Nov 02 21:29:09.638238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.638216 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d4d948381cf6fcb606ccf792670cb5b1579e780ae5071c99b60d76a68b3fbac" Nov 02 21:29:09.638341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.638085 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4" Nov 02 21:29:09.660341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.660315 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2sl\" (UniqueName: \"kubernetes.io/projected/240c5a07-0957-4f3e-94df-d35d7dfc420b-kube-api-access-xk2sl\") pod \"maas-api-revoke-sub-533db7e7-48kp5\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.672452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.672420 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:09.811017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:09.808827 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:10.095471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:10.095437 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5"] Nov 02 21:29:10.102027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:10.101995 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240c5a07_0957_4f3e_94df_d35d7dfc420b.slice/crio-96755de0666a94043f808a195ffdb39176b72c77c1a846fc73af53a4598402c7 WatchSource:0}: Error finding container 96755de0666a94043f808a195ffdb39176b72c77c1a846fc73af53a4598402c7: Status 404 returned error can't find the container with id 96755de0666a94043f808a195ffdb39176b72c77c1a846fc73af53a4598402c7 Nov 02 21:29:10.614309 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:29:10.614269 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:29:10.646051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:10.645952 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" event={"ID":"240c5a07-0957-4f3e-94df-d35d7dfc420b","Type":"ContainerStarted","Data":"6b15fa02a12f956ced4f79bffff513600cca5beb517c0338f667f987c3658edc"} Nov 02 21:29:10.646051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:10.646030 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" event={"ID":"240c5a07-0957-4f3e-94df-d35d7dfc420b","Type":"ContainerStarted","Data":"96755de0666a94043f808a195ffdb39176b72c77c1a846fc73af53a4598402c7"} Nov 02 21:29:10.753116 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:10.753058 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" podStartSLOduration=1.753038375 podStartE2EDuration="1.753038375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:10.752234007 +0000 UTC m=+980.307862339" watchObservedRunningTime="2026-11-02 21:29:10.753038375 +0000 UTC m=+980.308666737" Nov 02 21:29:11.654907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:11.654865 4441 generic.go:356] "Generic (PLEG): container finished" podID="240c5a07-0957-4f3e-94df-d35d7dfc420b" containerID="6b15fa02a12f956ced4f79bffff513600cca5beb517c0338f667f987c3658edc" exitCode=0 Nov 02 21:29:11.655325 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:11.654954 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" event={"ID":"240c5a07-0957-4f3e-94df-d35d7dfc420b","Type":"ContainerDied","Data":"6b15fa02a12f956ced4f79bffff513600cca5beb517c0338f667f987c3658edc"} Nov 02 21:29:12.798652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:12.798626 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:12.927399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:12.927325 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-openshift-ca\") pod \"240c5a07-0957-4f3e-94df-d35d7dfc420b\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " Nov 02 21:29:12.927563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:12.927445 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240c5a07-0957-4f3e-94df-d35d7dfc420b-kube-api-access-xk2sl\" (UniqueName: \"kubernetes.io/projected/240c5a07-0957-4f3e-94df-d35d7dfc420b-kube-api-access-xk2sl\") pod \"240c5a07-0957-4f3e-94df-d35d7dfc420b\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " Nov 02 21:29:12.927563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:12.927481 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-xks-ca\") pod \"240c5a07-0957-4f3e-94df-d35d7dfc420b\" (UID: \"240c5a07-0957-4f3e-94df-d35d7dfc420b\") " Nov 02 21:29:12.927735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:12.927712 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-openshift-ca" pod "240c5a07-0957-4f3e-94df-d35d7dfc420b" (UID: "240c5a07-0957-4f3e-94df-d35d7dfc420b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:12.929665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:12.929637 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-xks-ca" pod "240c5a07-0957-4f3e-94df-d35d7dfc420b" (UID: "240c5a07-0957-4f3e-94df-d35d7dfc420b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:12.929772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:12.929714 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240c5a07-0957-4f3e-94df-d35d7dfc420b-kube-api-access-xk2sl" pod "240c5a07-0957-4f3e-94df-d35d7dfc420b" (UID: "240c5a07-0957-4f3e-94df-d35d7dfc420b"). InnerVolumeSpecName "kube-api-access-xk2sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:13.028570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:13.028538 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2sl\" (UniqueName: \"kubernetes.io/projected/240c5a07-0957-4f3e-94df-d35d7dfc420b-kube-api-access-xk2sl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:13.028570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:13.028565 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:13.028570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:13.028574 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240c5a07-0957-4f3e-94df-d35d7dfc420b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:13.668590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:13.668558 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" Nov 02 21:29:13.668771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:13.668595 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5" event={"ID":"240c5a07-0957-4f3e-94df-d35d7dfc420b","Type":"ContainerDied","Data":"96755de0666a94043f808a195ffdb39176b72c77c1a846fc73af53a4598402c7"} Nov 02 21:29:13.668771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:13.668650 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96755de0666a94043f808a195ffdb39176b72c77c1a846fc73af53a4598402c7" Nov 02 21:29:14.307511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:14.307473 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:18.882617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:18.882575 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:23.415733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.415688 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:23.496009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.495961 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f"] Nov 02 21:29:23.496462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.496450 4441 cpu_manager.go:414] "containerMap: removing container" podUID="240c5a07-0957-4f3e-94df-d35d7dfc420b" containerName="revoke-keys" Nov 02 21:29:23.496524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.496463 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240c5a07-0957-4f3e-94df-d35d7dfc420b" containerName="revoke-keys" Nov 02 21:29:23.496561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.496548 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="240c5a07-0957-4f3e-94df-d35d7dfc420b" containerName="revoke-keys" Nov 02 21:29:23.499374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.499354 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.551649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.551624 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:29:23.552915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.552888 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f"] Nov 02 21:29:23.595628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.595604 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:29:23.595628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.595623 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:29:23.643932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.643909 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190f802f-8675-49bb-884b-65a57742f357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.644056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.643998 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbf94\" (UniqueName: \"kubernetes.io/projected/190f802f-8675-49bb-884b-65a57742f357-kube-api-access-mbf94\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.644056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.644029 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190f802f-8675-49bb-884b-65a57742f357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.649402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.649386 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:29:23.679280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.679214 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd"] Nov 02 21:29:23.682134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.682118 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.745072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.745046 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190f802f-8675-49bb-884b-65a57742f357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.745215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.745157 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbf94\" (UniqueName: \"kubernetes.io/projected/190f802f-8675-49bb-884b-65a57742f357-kube-api-access-mbf94\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.745215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.745203 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190f802f-8675-49bb-884b-65a57742f357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.747641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.747618 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190f802f-8675-49bb-884b-65a57742f357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.749283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.749261 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd"] Nov 02 21:29:23.749781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.749757 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190f802f-8675-49bb-884b-65a57742f357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.807911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.807876 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbf94\" (UniqueName: \"kubernetes.io/projected/190f802f-8675-49bb-884b-65a57742f357-kube-api-access-mbf94\") pod \"maas-api-revoke-sub-df54a85e-znk6f\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.811063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.811034 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:23.846329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.846294 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72ws\" (UniqueName: \"kubernetes.io/projected/7727674b-a53c-4ca5-a06e-98ddeecd5148-kube-api-access-c72ws\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.846470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.846359 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.846529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.846478 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.948070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.947960 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72ws\" (UniqueName: \"kubernetes.io/projected/7727674b-a53c-4ca5-a06e-98ddeecd5148-kube-api-access-c72ws\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.948070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.948061 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.948291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.948095 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.948914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.948889 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:23.950681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:23.950657 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:24.001923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.001893 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72ws\" (UniqueName: \"kubernetes.io/projected/7727674b-a53c-4ca5-a06e-98ddeecd5148-kube-api-access-c72ws\") pod \"maas-api-revoke-sub-fad6d08a-tdlmd\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:24.189809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.189773 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f"] Nov 02 21:29:24.195924 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:24.195895 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190f802f_8675_49bb_884b_65a57742f357.slice/crio-652e0cca13910006102de69d73d3bf5090151ec793a7f7ecd90fef40003315ae WatchSource:0}: Error finding container 652e0cca13910006102de69d73d3bf5090151ec793a7f7ecd90fef40003315ae: Status 404 returned error can't find the container with id 652e0cca13910006102de69d73d3bf5090151ec793a7f7ecd90fef40003315ae Nov 02 21:29:24.294324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.294297 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:24.664286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.664252 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd"] Nov 02 21:29:24.669731 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:24.669705 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7727674b_a53c_4ca5_a06e_98ddeecd5148.slice/crio-84ae0675694c23c055caf556f918ec37f78775f6706b520e2df1071758d36c6d WatchSource:0}: Error finding container 84ae0675694c23c055caf556f918ec37f78775f6706b520e2df1071758d36c6d: Status 404 returned error can't find the container with id 84ae0675694c23c055caf556f918ec37f78775f6706b520e2df1071758d36c6d Nov 02 21:29:24.749249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.749218 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" event={"ID":"190f802f-8675-49bb-884b-65a57742f357","Type":"ContainerStarted","Data":"ac9f03f03552ee560b9089d45760a3258badb1d81c36416d56738e82177f0521"} Nov 02 21:29:24.749433 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.749285 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" event={"ID":"190f802f-8675-49bb-884b-65a57742f357","Type":"ContainerStarted","Data":"652e0cca13910006102de69d73d3bf5090151ec793a7f7ecd90fef40003315ae"} Nov 02 21:29:24.750391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.750358 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" event={"ID":"7727674b-a53c-4ca5-a06e-98ddeecd5148","Type":"ContainerStarted","Data":"84ae0675694c23c055caf556f918ec37f78775f6706b520e2df1071758d36c6d"} Nov 02 21:29:24.860377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:24.860260 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" podStartSLOduration=1.860241276 podStartE2EDuration="1.860241276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:24.855690654 +0000 UTC m=+994.411318990" watchObservedRunningTime="2026-11-02 21:29:24.860241276 +0000 UTC m=+994.415869614" Nov 02 21:29:25.758594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:25.758555 4441 generic.go:356] "Generic (PLEG): container finished" podID="190f802f-8675-49bb-884b-65a57742f357" containerID="ac9f03f03552ee560b9089d45760a3258badb1d81c36416d56738e82177f0521" exitCode=0 Nov 02 21:29:25.759002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:25.758642 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" event={"ID":"190f802f-8675-49bb-884b-65a57742f357","Type":"ContainerDied","Data":"ac9f03f03552ee560b9089d45760a3258badb1d81c36416d56738e82177f0521"} Nov 02 21:29:25.764125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:25.764099 4441 generic.go:356] "Generic (PLEG): container finished" podID="7727674b-a53c-4ca5-a06e-98ddeecd5148" containerID="919400fbf8aa9e4f9f5c68bbfdfbdfb231ff622a3d32a0395771920bc72953df" exitCode=0 Nov 02 21:29:25.764258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:25.764189 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" event={"ID":"7727674b-a53c-4ca5-a06e-98ddeecd5148","Type":"ContainerDied","Data":"919400fbf8aa9e4f9f5c68bbfdfbdfb231ff622a3d32a0395771920bc72953df"} Nov 02 21:29:26.286434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.286402 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8"] Nov 02 21:29:26.289369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.289348 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.345017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.344989 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8"] Nov 02 21:29:26.370134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.370104 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.370285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.370145 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.370352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.370310 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k952l\" (UniqueName: \"kubernetes.io/projected/299d18d5-42dc-4eb5-990e-c069d11a414e-kube-api-access-k952l\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.470937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.470905 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.470937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.470935 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.471187 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.471010 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k952l\" (UniqueName: \"kubernetes.io/projected/299d18d5-42dc-4eb5-990e-c069d11a414e-kube-api-access-k952l\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.471713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.471690 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.473435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.473409 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.524540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.524513 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k952l\" (UniqueName: \"kubernetes.io/projected/299d18d5-42dc-4eb5-990e-c069d11a414e-kube-api-access-k952l\") pod \"maas-api-revoke-sub-99bab561-66hn8\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.600997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.600963 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:26.714007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.713957 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:26.922985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.922935 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:26.937370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.937345 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:26.977589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.977565 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-xks-ca\") pod \"7727674b-a53c-4ca5-a06e-98ddeecd5148\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " Nov 02 21:29:26.977724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.977620 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190f802f-8675-49bb-884b-65a57742f357-kube-api-access-mbf94\" (UniqueName: \"kubernetes.io/projected/190f802f-8675-49bb-884b-65a57742f357-kube-api-access-mbf94\") pod \"190f802f-8675-49bb-884b-65a57742f357\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " Nov 02 21:29:26.977724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.977649 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190f802f-8675-49bb-884b-65a57742f357-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190f802f-8675-49bb-884b-65a57742f357-maas-api-xks-ca\") pod \"190f802f-8675-49bb-884b-65a57742f357\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " Nov 02 21:29:26.977853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.977759 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190f802f-8675-49bb-884b-65a57742f357-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190f802f-8675-49bb-884b-65a57742f357-maas-api-openshift-ca\") pod \"190f802f-8675-49bb-884b-65a57742f357\" (UID: \"190f802f-8675-49bb-884b-65a57742f357\") " Nov 02 21:29:26.977853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.977823 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7727674b-a53c-4ca5-a06e-98ddeecd5148-kube-api-access-c72ws\" (UniqueName: \"kubernetes.io/projected/7727674b-a53c-4ca5-a06e-98ddeecd5148-kube-api-access-c72ws\") pod \"7727674b-a53c-4ca5-a06e-98ddeecd5148\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " Nov 02 21:29:26.977953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.977888 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-openshift-ca\") pod \"7727674b-a53c-4ca5-a06e-98ddeecd5148\" (UID: \"7727674b-a53c-4ca5-a06e-98ddeecd5148\") " Nov 02 21:29:26.978252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.978225 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190f802f-8675-49bb-884b-65a57742f357-maas-api-openshift-ca" pod "190f802f-8675-49bb-884b-65a57742f357" (UID: "190f802f-8675-49bb-884b-65a57742f357"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:26.978552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.978520 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-openshift-ca" pod "7727674b-a53c-4ca5-a06e-98ddeecd5148" (UID: "7727674b-a53c-4ca5-a06e-98ddeecd5148"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:26.980637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.980615 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190f802f-8675-49bb-884b-65a57742f357-maas-api-xks-ca" pod "190f802f-8675-49bb-884b-65a57742f357" (UID: "190f802f-8675-49bb-884b-65a57742f357"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:26.980936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.980904 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190f802f-8675-49bb-884b-65a57742f357-kube-api-access-mbf94" pod "190f802f-8675-49bb-884b-65a57742f357" (UID: "190f802f-8675-49bb-884b-65a57742f357"). InnerVolumeSpecName "kube-api-access-mbf94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:26.980936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.980930 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-xks-ca" pod "7727674b-a53c-4ca5-a06e-98ddeecd5148" (UID: "7727674b-a53c-4ca5-a06e-98ddeecd5148"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:26.982103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.982079 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7727674b-a53c-4ca5-a06e-98ddeecd5148-kube-api-access-c72ws" pod "7727674b-a53c-4ca5-a06e-98ddeecd5148" (UID: "7727674b-a53c-4ca5-a06e-98ddeecd5148"). InnerVolumeSpecName "kube-api-access-c72ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:26.996210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:26.996171 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8"] Nov 02 21:29:27.079392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.079365 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190f802f-8675-49bb-884b-65a57742f357-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:27.079392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.079391 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190f802f-8675-49bb-884b-65a57742f357-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:27.079615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.079401 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72ws\" (UniqueName: \"kubernetes.io/projected/7727674b-a53c-4ca5-a06e-98ddeecd5148-kube-api-access-c72ws\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:27.079615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.079412 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:27.079615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.079420 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7727674b-a53c-4ca5-a06e-98ddeecd5148-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:27.079615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.079428 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbf94\" (UniqueName: \"kubernetes.io/projected/190f802f-8675-49bb-884b-65a57742f357-kube-api-access-mbf94\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:27.284110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.284016 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:27.641801 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.641769 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf"] Nov 02 21:29:27.642277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.642263 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7727674b-a53c-4ca5-a06e-98ddeecd5148" containerName="revoke-keys" Nov 02 21:29:27.642332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.642277 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7727674b-a53c-4ca5-a06e-98ddeecd5148" containerName="revoke-keys" Nov 02 21:29:27.642332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.642299 4441 cpu_manager.go:414] "containerMap: removing container" podUID="190f802f-8675-49bb-884b-65a57742f357" containerName="revoke-keys" Nov 02 21:29:27.642332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.642304 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190f802f-8675-49bb-884b-65a57742f357" containerName="revoke-keys" Nov 02 21:29:27.642428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.642410 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7727674b-a53c-4ca5-a06e-98ddeecd5148" containerName="revoke-keys" Nov 02 21:29:27.642428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.642421 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="190f802f-8675-49bb-884b-65a57742f357" containerName="revoke-keys" Nov 02 21:29:27.644900 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.644877 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.686598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.686568 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.686733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.686648 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.686782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.686760 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2x24\" (UniqueName: \"kubernetes.io/projected/6ad23875-d5e3-45e2-b608-3747f5879be0-kube-api-access-l2x24\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.698738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.698708 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf"] Nov 02 21:29:27.778511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.778475 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" event={"ID":"299d18d5-42dc-4eb5-990e-c069d11a414e","Type":"ContainerStarted","Data":"78e49ffba4f2a6f58df156bdfeeb6d0c6df3c21dc11bb4af33fb9bf0b56bed89"} Nov 02 21:29:27.778667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.778532 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" event={"ID":"299d18d5-42dc-4eb5-990e-c069d11a414e","Type":"ContainerStarted","Data":"122b608c207fb3bf800eb8fc940837d236480691a6c73f579ad49157a8308fb4"} Nov 02 21:29:27.779832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.779809 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" Nov 02 21:29:27.779940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.779851 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f" event={"ID":"190f802f-8675-49bb-884b-65a57742f357","Type":"ContainerDied","Data":"652e0cca13910006102de69d73d3bf5090151ec793a7f7ecd90fef40003315ae"} Nov 02 21:29:27.779940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.779900 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="652e0cca13910006102de69d73d3bf5090151ec793a7f7ecd90fef40003315ae" Nov 02 21:29:27.781252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.781228 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" Nov 02 21:29:27.781345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.781268 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd" event={"ID":"7727674b-a53c-4ca5-a06e-98ddeecd5148","Type":"ContainerDied","Data":"84ae0675694c23c055caf556f918ec37f78775f6706b520e2df1071758d36c6d"} Nov 02 21:29:27.781345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.781303 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ae0675694c23c055caf556f918ec37f78775f6706b520e2df1071758d36c6d" Nov 02 21:29:27.787351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.787319 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.787450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.787394 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.787513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.787481 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2x24\" (UniqueName: \"kubernetes.io/projected/6ad23875-d5e3-45e2-b608-3747f5879be0-kube-api-access-l2x24\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.788161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.788142 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.790205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.790180 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.841002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.840958 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2x24\" (UniqueName: \"kubernetes.io/projected/6ad23875-d5e3-45e2-b608-3747f5879be0-kube-api-access-l2x24\") pod \"maas-api-revoke-sub-18ad616d-825nf\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:27.887765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.887721 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" podStartSLOduration=1.887709179 podStartE2EDuration="1.887709179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:27.884456618 +0000 UTC m=+997.440084957" watchObservedRunningTime="2026-11-02 21:29:27.887709179 +0000 UTC m=+997.443337517" Nov 02 21:29:27.906747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.906691 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:27.959209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:27.959183 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:28.339699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:28.339663 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf"] Nov 02 21:29:28.344600 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:28.344574 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad23875_d5e3_45e2_b608_3747f5879be0.slice/crio-b4de33e1dff5b65a74168dfb90e1d02efcb39d8ccbeed661711a7d5ace6a6c6b WatchSource:0}: Error finding container b4de33e1dff5b65a74168dfb90e1d02efcb39d8ccbeed661711a7d5ace6a6c6b: Status 404 returned error can't find the container with id b4de33e1dff5b65a74168dfb90e1d02efcb39d8ccbeed661711a7d5ace6a6c6b Nov 02 21:29:28.791879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:28.791845 4441 generic.go:356] "Generic (PLEG): container finished" podID="299d18d5-42dc-4eb5-990e-c069d11a414e" containerID="78e49ffba4f2a6f58df156bdfeeb6d0c6df3c21dc11bb4af33fb9bf0b56bed89" exitCode=0 Nov 02 21:29:28.792050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:28.791941 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" event={"ID":"299d18d5-42dc-4eb5-990e-c069d11a414e","Type":"ContainerDied","Data":"78e49ffba4f2a6f58df156bdfeeb6d0c6df3c21dc11bb4af33fb9bf0b56bed89"} Nov 02 21:29:28.793908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:28.793881 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" event={"ID":"6ad23875-d5e3-45e2-b608-3747f5879be0","Type":"ContainerStarted","Data":"a033b930fb4080abaf1255b3e0f4ac9a74fbf8e4f39ea8789b4e6f6bc127ad4e"} Nov 02 21:29:28.794048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:28.793923 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" event={"ID":"6ad23875-d5e3-45e2-b608-3747f5879be0","Type":"ContainerStarted","Data":"b4de33e1dff5b65a74168dfb90e1d02efcb39d8ccbeed661711a7d5ace6a6c6b"} Nov 02 21:29:29.009761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:29.009656 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" podStartSLOduration=2.009641406 podStartE2EDuration="2.009641406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:29.007004273 +0000 UTC m=+998.562632610" watchObservedRunningTime="2026-11-02 21:29:29.009641406 +0000 UTC m=+998.565269768" Nov 02 21:29:29.803745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:29.803702 4441 generic.go:356] "Generic (PLEG): container finished" podID="6ad23875-d5e3-45e2-b608-3747f5879be0" containerID="a033b930fb4080abaf1255b3e0f4ac9a74fbf8e4f39ea8789b4e6f6bc127ad4e" exitCode=0 Nov 02 21:29:29.803916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:29.803793 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" event={"ID":"6ad23875-d5e3-45e2-b608-3747f5879be0","Type":"ContainerDied","Data":"a033b930fb4080abaf1255b3e0f4ac9a74fbf8e4f39ea8789b4e6f6bc127ad4e"} Nov 02 21:29:29.952876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:29.952853 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:30.014367 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.014333 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299d18d5-42dc-4eb5-990e-c069d11a414e-kube-api-access-k952l\" (UniqueName: \"kubernetes.io/projected/299d18d5-42dc-4eb5-990e-c069d11a414e-kube-api-access-k952l\") pod \"299d18d5-42dc-4eb5-990e-c069d11a414e\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " Nov 02 21:29:30.014815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.014561 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-xks-ca\") pod \"299d18d5-42dc-4eb5-990e-c069d11a414e\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " Nov 02 21:29:30.014815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.014618 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-openshift-ca\") pod \"299d18d5-42dc-4eb5-990e-c069d11a414e\" (UID: \"299d18d5-42dc-4eb5-990e-c069d11a414e\") " Nov 02 21:29:30.015139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.015104 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-openshift-ca" pod "299d18d5-42dc-4eb5-990e-c069d11a414e" (UID: "299d18d5-42dc-4eb5-990e-c069d11a414e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:30.016582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.016551 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299d18d5-42dc-4eb5-990e-c069d11a414e-kube-api-access-k952l" pod "299d18d5-42dc-4eb5-990e-c069d11a414e" (UID: "299d18d5-42dc-4eb5-990e-c069d11a414e"). InnerVolumeSpecName "kube-api-access-k952l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:30.016676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.016629 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-xks-ca" pod "299d18d5-42dc-4eb5-990e-c069d11a414e" (UID: "299d18d5-42dc-4eb5-990e-c069d11a414e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:30.116111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.116084 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:30.116111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.116110 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299d18d5-42dc-4eb5-990e-c069d11a414e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:30.116354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.116120 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k952l\" (UniqueName: \"kubernetes.io/projected/299d18d5-42dc-4eb5-990e-c069d11a414e-kube-api-access-k952l\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:30.813075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.813037 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" event={"ID":"299d18d5-42dc-4eb5-990e-c069d11a414e","Type":"ContainerDied","Data":"122b608c207fb3bf800eb8fc940837d236480691a6c73f579ad49157a8308fb4"} Nov 02 21:29:30.813245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.813080 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8" Nov 02 21:29:30.813245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.813096 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="122b608c207fb3bf800eb8fc940837d236480691a6c73f579ad49157a8308fb4" Nov 02 21:29:30.955873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:30.955847 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:31.027552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.027526 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-xks-ca\") pod \"6ad23875-d5e3-45e2-b608-3747f5879be0\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " Nov 02 21:29:31.028006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.027594 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad23875-d5e3-45e2-b608-3747f5879be0-kube-api-access-l2x24\" (UniqueName: \"kubernetes.io/projected/6ad23875-d5e3-45e2-b608-3747f5879be0-kube-api-access-l2x24\") pod \"6ad23875-d5e3-45e2-b608-3747f5879be0\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " Nov 02 21:29:31.028006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.027655 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-openshift-ca\") pod \"6ad23875-d5e3-45e2-b608-3747f5879be0\" (UID: \"6ad23875-d5e3-45e2-b608-3747f5879be0\") " Nov 02 21:29:31.028157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.028131 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-openshift-ca" pod "6ad23875-d5e3-45e2-b608-3747f5879be0" (UID: "6ad23875-d5e3-45e2-b608-3747f5879be0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:31.029627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.029604 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-xks-ca" pod "6ad23875-d5e3-45e2-b608-3747f5879be0" (UID: "6ad23875-d5e3-45e2-b608-3747f5879be0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:31.029695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.029641 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad23875-d5e3-45e2-b608-3747f5879be0-kube-api-access-l2x24" pod "6ad23875-d5e3-45e2-b608-3747f5879be0" (UID: "6ad23875-d5e3-45e2-b608-3747f5879be0"). InnerVolumeSpecName "kube-api-access-l2x24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:31.129057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.128952 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2x24\" (UniqueName: \"kubernetes.io/projected/6ad23875-d5e3-45e2-b608-3747f5879be0-kube-api-access-l2x24\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:31.129057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.129001 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:31.129057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.129012 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23875-d5e3-45e2-b608-3747f5879be0-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:31.130579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.130549 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:31.249902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.249866 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:31.824138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.824104 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" Nov 02 21:29:31.824346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.824136 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf" event={"ID":"6ad23875-d5e3-45e2-b608-3747f5879be0","Type":"ContainerDied","Data":"b4de33e1dff5b65a74168dfb90e1d02efcb39d8ccbeed661711a7d5ace6a6c6b"} Nov 02 21:29:31.824346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:31.824186 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4de33e1dff5b65a74168dfb90e1d02efcb39d8ccbeed661711a7d5ace6a6c6b" Nov 02 21:29:32.490989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:32.490941 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:37.077197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:37.077162 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:39.126290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.126256 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7"] Nov 02 21:29:39.126735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.126721 4441 cpu_manager.go:414] "containerMap: removing container" podUID="299d18d5-42dc-4eb5-990e-c069d11a414e" containerName="revoke-keys" Nov 02 21:29:39.126787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.126736 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299d18d5-42dc-4eb5-990e-c069d11a414e" containerName="revoke-keys" Nov 02 21:29:39.126787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.126755 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6ad23875-d5e3-45e2-b608-3747f5879be0" containerName="revoke-keys" Nov 02 21:29:39.126787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.126759 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad23875-d5e3-45e2-b608-3747f5879be0" containerName="revoke-keys" Nov 02 21:29:39.126876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.126838 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad23875-d5e3-45e2-b608-3747f5879be0" containerName="revoke-keys" Nov 02 21:29:39.126876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.126852 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="299d18d5-42dc-4eb5-990e-c069d11a414e" containerName="revoke-keys" Nov 02 21:29:39.130171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.130149 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.182658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.182631 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7"] Nov 02 21:29:39.218195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.218164 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptsm\" (UniqueName: \"kubernetes.io/projected/1c986c51-4b88-4483-b10f-a5c30e55f32b-kube-api-access-kptsm\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.218320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.218203 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.218373 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.218323 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.225171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.225142 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:29:39.225171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.225143 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:29:39.225277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.225184 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:29:39.226788 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.226772 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:29:39.319172 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.319142 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptsm\" (UniqueName: \"kubernetes.io/projected/1c986c51-4b88-4483-b10f-a5c30e55f32b-kube-api-access-kptsm\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.319329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.319185 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.319329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.319267 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.319989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.319950 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.321503 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.321487 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.372369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.372340 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptsm\" (UniqueName: \"kubernetes.io/projected/1c986c51-4b88-4483-b10f-a5c30e55f32b-kube-api-access-kptsm\") pod \"maas-api-revoke-sub-e8ff4cc5-5qrq7\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.443652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.443594 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:39.815699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.815667 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7"] Nov 02 21:29:39.821320 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:39.821290 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c986c51_4b88_4483_b10f_a5c30e55f32b.slice/crio-5e77291fadf269b13e5018d38eef671635042e4ae76ac03267b1f5e6f7c64314 WatchSource:0}: Error finding container 5e77291fadf269b13e5018d38eef671635042e4ae76ac03267b1f5e6f7c64314: Status 404 returned error can't find the container with id 5e77291fadf269b13e5018d38eef671635042e4ae76ac03267b1f5e6f7c64314 Nov 02 21:29:39.881640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:39.881607 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" event={"ID":"1c986c51-4b88-4483-b10f-a5c30e55f32b","Type":"ContainerStarted","Data":"5e77291fadf269b13e5018d38eef671635042e4ae76ac03267b1f5e6f7c64314"} Nov 02 21:29:40.614843 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:29:40.614802 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:29:40.890504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:40.890419 4441 generic.go:356] "Generic (PLEG): container finished" podID="1c986c51-4b88-4483-b10f-a5c30e55f32b" containerID="ad93625c60fef6a87dcd783546c6f1583b6f9d2da3761bbcb78ea3c8da63c65c" exitCode=0 Nov 02 21:29:40.890504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:40.890491 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" event={"ID":"1c986c51-4b88-4483-b10f-a5c30e55f32b","Type":"ContainerDied","Data":"ad93625c60fef6a87dcd783546c6f1583b6f9d2da3761bbcb78ea3c8da63c65c"} Nov 02 21:29:41.047087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.047054 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9"] Nov 02 21:29:41.050792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.050773 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.103450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.103423 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9"] Nov 02 21:29:41.138690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.138661 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.138822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.138726 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbg8\" (UniqueName: \"kubernetes.io/projected/c658aee1-1ef6-4bcf-8430-c9922bc18c37-kube-api-access-dtbg8\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.138910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.138886 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.240240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.240169 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.240240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.240229 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbg8\" (UniqueName: \"kubernetes.io/projected/c658aee1-1ef6-4bcf-8430-c9922bc18c37-kube-api-access-dtbg8\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.240416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.240282 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.240945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.240926 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.242403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.242384 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.295706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.295675 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbg8\" (UniqueName: \"kubernetes.io/projected/c658aee1-1ef6-4bcf-8430-c9922bc18c37-kube-api-access-dtbg8\") pod \"maas-api-revoke-sub-d0bcc74a-bhgp9\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.365837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.365808 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:41.499446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.499377 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:41.737779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.737745 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9"] Nov 02 21:29:41.742870 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:41.742845 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc658aee1_1ef6_4bcf_8430_c9922bc18c37.slice/crio-dd2da4d321aaf62e0056bfcbff23d51c5ce8408fb870abf3ee34b475b01c2036 WatchSource:0}: Error finding container dd2da4d321aaf62e0056bfcbff23d51c5ce8408fb870abf3ee34b475b01c2036: Status 404 returned error can't find the container with id dd2da4d321aaf62e0056bfcbff23d51c5ce8408fb870abf3ee34b475b01c2036 Nov 02 21:29:41.898242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:41.898212 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" event={"ID":"c658aee1-1ef6-4bcf-8430-c9922bc18c37","Type":"ContainerStarted","Data":"dd2da4d321aaf62e0056bfcbff23d51c5ce8408fb870abf3ee34b475b01c2036"} Nov 02 21:29:42.046412 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.046388 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:42.149041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.149014 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-xks-ca\") pod \"1c986c51-4b88-4483-b10f-a5c30e55f32b\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " Nov 02 21:29:42.149188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.149115 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-openshift-ca\") pod \"1c986c51-4b88-4483-b10f-a5c30e55f32b\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " Nov 02 21:29:42.149188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.149161 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c986c51-4b88-4483-b10f-a5c30e55f32b-kube-api-access-kptsm\" (UniqueName: \"kubernetes.io/projected/1c986c51-4b88-4483-b10f-a5c30e55f32b-kube-api-access-kptsm\") pod \"1c986c51-4b88-4483-b10f-a5c30e55f32b\" (UID: \"1c986c51-4b88-4483-b10f-a5c30e55f32b\") " Nov 02 21:29:42.149474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.149442 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-openshift-ca" pod "1c986c51-4b88-4483-b10f-a5c30e55f32b" (UID: "1c986c51-4b88-4483-b10f-a5c30e55f32b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:42.149612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.149554 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:42.151277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.151252 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-xks-ca" pod "1c986c51-4b88-4483-b10f-a5c30e55f32b" (UID: "1c986c51-4b88-4483-b10f-a5c30e55f32b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:42.151367 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.151303 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c986c51-4b88-4483-b10f-a5c30e55f32b-kube-api-access-kptsm" pod "1c986c51-4b88-4483-b10f-a5c30e55f32b" (UID: "1c986c51-4b88-4483-b10f-a5c30e55f32b"). InnerVolumeSpecName "kube-api-access-kptsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:42.250861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.250829 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c986c51-4b88-4483-b10f-a5c30e55f32b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:42.250861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.250858 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptsm\" (UniqueName: \"kubernetes.io/projected/1c986c51-4b88-4483-b10f-a5c30e55f32b-kube-api-access-kptsm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:42.516326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.516287 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv"] Nov 02 21:29:42.517117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.517095 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1c986c51-4b88-4483-b10f-a5c30e55f32b" containerName="revoke-keys" Nov 02 21:29:42.517173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.517121 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c986c51-4b88-4483-b10f-a5c30e55f32b" containerName="revoke-keys" Nov 02 21:29:42.517255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.517242 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c986c51-4b88-4483-b10f-a5c30e55f32b" containerName="revoke-keys" Nov 02 21:29:42.520693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.520662 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.572291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.572252 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv"] Nov 02 21:29:42.655539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.655472 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwkr\" (UniqueName: \"kubernetes.io/projected/224635dd-b127-4f40-a895-3cc532404a4f-kube-api-access-wbwkr\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.655539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.655525 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.655757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.655657 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.757100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.757073 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.757501 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.757130 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.757501 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.757276 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwkr\" (UniqueName: \"kubernetes.io/projected/224635dd-b127-4f40-a895-3cc532404a4f-kube-api-access-wbwkr\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.757837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.757812 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.759319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.759299 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.812751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.812719 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwkr\" (UniqueName: \"kubernetes.io/projected/224635dd-b127-4f40-a895-3cc532404a4f-kube-api-access-wbwkr\") pod \"maas-api-revoke-sub-c680b95e-rqdzv\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.835174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.835142 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:42.935739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.935655 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" event={"ID":"1c986c51-4b88-4483-b10f-a5c30e55f32b","Type":"ContainerDied","Data":"5e77291fadf269b13e5018d38eef671635042e4ae76ac03267b1f5e6f7c64314"} Nov 02 21:29:42.935878 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.935763 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e77291fadf269b13e5018d38eef671635042e4ae76ac03267b1f5e6f7c64314" Nov 02 21:29:42.935944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.935890 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7" Nov 02 21:29:42.941499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.941472 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" event={"ID":"c658aee1-1ef6-4bcf-8430-c9922bc18c37","Type":"ContainerDied","Data":"236a3ec0992281c0f818150aaff59f92269874c30d73f0e061c53274d55bd6dc"} Nov 02 21:29:42.941615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:42.941471 4441 generic.go:356] "Generic (PLEG): container finished" podID="c658aee1-1ef6-4bcf-8430-c9922bc18c37" containerID="236a3ec0992281c0f818150aaff59f92269874c30d73f0e061c53274d55bd6dc" exitCode=0 Nov 02 21:29:43.218381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.218147 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv"] Nov 02 21:29:43.221218 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:43.221185 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224635dd_b127_4f40_a895_3cc532404a4f.slice/crio-4abbdbe0eedd20cff6a09fccba1cca7f6f3351cbb1f5ec01260cef36b22b53c2 WatchSource:0}: Error finding container 4abbdbe0eedd20cff6a09fccba1cca7f6f3351cbb1f5ec01260cef36b22b53c2: Status 404 returned error can't find the container with id 4abbdbe0eedd20cff6a09fccba1cca7f6f3351cbb1f5ec01260cef36b22b53c2 Nov 02 21:29:43.772649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.772553 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z"] Nov 02 21:29:43.776271 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.776250 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.830300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.830267 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"openshift-service-ca.crt\"" Nov 02 21:29:43.830473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.830285 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"kube-root-ca.crt\"" Nov 02 21:29:43.830473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.830377 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"default-dockercfg-ptsrn\"" Nov 02 21:29:43.830473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.830458 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"distinct-w3-9a6cf2-kserve-self-signed-certs\"" Nov 02 21:29:43.831871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.831846 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z"] Nov 02 21:29:43.863480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.863454 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:43.869358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.869336 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69caa72-3429-40e8-886e-0a97015ef6e9-tls-certs\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.869467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.869418 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-model-cache\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.869467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.869454 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-home\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.869571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.869536 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5r6v\" (UniqueName: \"kubernetes.io/projected/f69caa72-3429-40e8-886e-0a97015ef6e9-kube-api-access-d5r6v\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.869571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.869561 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-tmp-dir\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.869727 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.869704 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-dshm\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.950261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.950223 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" event={"ID":"224635dd-b127-4f40-a895-3cc532404a4f","Type":"ContainerStarted","Data":"0ae11528cf995d7c87430540434b29e10d7c050458cd829632842fe412451546"} Nov 02 21:29:43.950427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.950291 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" event={"ID":"224635dd-b127-4f40-a895-3cc532404a4f","Type":"ContainerStarted","Data":"4abbdbe0eedd20cff6a09fccba1cca7f6f3351cbb1f5ec01260cef36b22b53c2"} Nov 02 21:29:43.970861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.970830 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69caa72-3429-40e8-886e-0a97015ef6e9-tls-certs\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.970870 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-model-cache\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.970897 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-home\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.970956 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5r6v\" (UniqueName: \"kubernetes.io/projected/f69caa72-3429-40e8-886e-0a97015ef6e9-kube-api-access-d5r6v\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971165 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.971008 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-tmp-dir\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971165 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.971095 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-dshm\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.971311 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-model-cache\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.971343 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-home\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.971570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.971491 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-tmp-dir\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.973402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.973374 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-dshm\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:43.973673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:43.973654 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69caa72-3429-40e8-886e-0a97015ef6e9-tls-certs\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:44.024885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.024819 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5r6v\" (UniqueName: \"kubernetes.io/projected/f69caa72-3429-40e8-886e-0a97015ef6e9-kube-api-access-d5r6v\") pod \"distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:44.061168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.061107 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" podStartSLOduration=2.061089353 podStartE2EDuration="2.061089353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:44.05913395 +0000 UTC m=+1013.614762292" watchObservedRunningTime="2026-11-02 21:29:44.061089353 +0000 UTC m=+1013.616717692" Nov 02 21:29:44.089331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.089306 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:44.149756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.149729 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:44.279147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.279061 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-openshift-ca\") pod \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " Nov 02 21:29:44.279316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.279175 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-xks-ca\") pod \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " Nov 02 21:29:44.279316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.279300 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c658aee1-1ef6-4bcf-8430-c9922bc18c37-kube-api-access-dtbg8\" (UniqueName: \"kubernetes.io/projected/c658aee1-1ef6-4bcf-8430-c9922bc18c37-kube-api-access-dtbg8\") pod \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\" (UID: \"c658aee1-1ef6-4bcf-8430-c9922bc18c37\") " Nov 02 21:29:44.279432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.279392 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-openshift-ca" pod "c658aee1-1ef6-4bcf-8430-c9922bc18c37" (UID: "c658aee1-1ef6-4bcf-8430-c9922bc18c37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:44.279652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.279635 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:44.281262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.281230 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-xks-ca" pod "c658aee1-1ef6-4bcf-8430-c9922bc18c37" (UID: "c658aee1-1ef6-4bcf-8430-c9922bc18c37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:44.281394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.281360 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c658aee1-1ef6-4bcf-8430-c9922bc18c37-kube-api-access-dtbg8" pod "c658aee1-1ef6-4bcf-8430-c9922bc18c37" (UID: "c658aee1-1ef6-4bcf-8430-c9922bc18c37"). InnerVolumeSpecName "kube-api-access-dtbg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:44.380587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.380559 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c658aee1-1ef6-4bcf-8430-c9922bc18c37-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:44.380587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.380584 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtbg8\" (UniqueName: \"kubernetes.io/projected/c658aee1-1ef6-4bcf-8430-c9922bc18c37-kube-api-access-dtbg8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:44.464502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.464466 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z"] Nov 02 21:29:44.468117 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:44.468085 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf69caa72_3429_40e8_886e_0a97015ef6e9.slice/crio-68927e1c9637276510b9b26194af561b3e38b590b5e1b11707be403ea60f2e96 WatchSource:0}: Error finding container 68927e1c9637276510b9b26194af561b3e38b590b5e1b11707be403ea60f2e96: Status 404 returned error can't find the container with id 68927e1c9637276510b9b26194af561b3e38b590b5e1b11707be403ea60f2e96 Nov 02 21:29:44.961348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.961311 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" event={"ID":"f69caa72-3429-40e8-886e-0a97015ef6e9","Type":"ContainerStarted","Data":"d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5"} Nov 02 21:29:44.961780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.961366 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" event={"ID":"f69caa72-3429-40e8-886e-0a97015ef6e9","Type":"ContainerStarted","Data":"68927e1c9637276510b9b26194af561b3e38b590b5e1b11707be403ea60f2e96"} Nov 02 21:29:44.963016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.962993 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" Nov 02 21:29:44.963016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.963001 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9" event={"ID":"c658aee1-1ef6-4bcf-8430-c9922bc18c37","Type":"ContainerDied","Data":"dd2da4d321aaf62e0056bfcbff23d51c5ce8408fb870abf3ee34b475b01c2036"} Nov 02 21:29:44.963235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.963048 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd2da4d321aaf62e0056bfcbff23d51c5ce8408fb870abf3ee34b475b01c2036" Nov 02 21:29:44.964770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.964734 4441 generic.go:356] "Generic (PLEG): container finished" podID="224635dd-b127-4f40-a895-3cc532404a4f" containerID="0ae11528cf995d7c87430540434b29e10d7c050458cd829632842fe412451546" exitCode=0 Nov 02 21:29:44.964770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:44.964769 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" event={"ID":"224635dd-b127-4f40-a895-3cc532404a4f","Type":"ContainerDied","Data":"0ae11528cf995d7c87430540434b29e10d7c050458cd829632842fe412451546"} Nov 02 21:29:45.077320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.077050 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podStartSLOduration=2.077033991 podStartE2EDuration="2.077033991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:45.074305998 +0000 UTC m=+1014.629934342" watchObservedRunningTime="2026-11-02 21:29:45.077033991 +0000 UTC m=+1014.632662330" Nov 02 21:29:45.090422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.090396 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:45.092105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.092080 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z"] Nov 02 21:29:45.101675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.101646 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c658aee1-1ef6-4bcf-8430-c9922bc18c37" containerName="revoke-keys" Nov 02 21:29:45.101675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.101672 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c658aee1-1ef6-4bcf-8430-c9922bc18c37" containerName="revoke-keys" Nov 02 21:29:45.101891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.101814 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c658aee1-1ef6-4bcf-8430-c9922bc18c37" containerName="revoke-keys" Nov 02 21:29:45.105437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.105418 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.182215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.182090 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z"] Nov 02 21:29:45.189433 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.189386 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.189597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.189455 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmtz\" (UniqueName: \"kubernetes.io/projected/b7cb0869-4634-46af-b380-c1d04cae724a-kube-api-access-8nmtz\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.189597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.189491 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.290594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.290504 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.290594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.290569 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmtz\" (UniqueName: \"kubernetes.io/projected/b7cb0869-4634-46af-b380-c1d04cae724a-kube-api-access-8nmtz\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.290787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.290608 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.291536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.291507 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.293185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.293161 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.308019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.307992 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg"] Nov 02 21:29:45.312847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.312828 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.344220 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.344199 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmtz\" (UniqueName: \"kubernetes.io/projected/b7cb0869-4634-46af-b380-c1d04cae724a-kube-api-access-8nmtz\") pod \"maas-api-revoke-sub-811430bc-m9m7z\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.362383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.362357 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9a6cf2-openshift-default-dockercfg-d2zhq\"" Nov 02 21:29:45.364276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.364251 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg"] Nov 02 21:29:45.391685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.391634 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nvp\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-kube-api-access-f5nvp\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.391685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.391691 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26245fbe-359b-4ff6-b6b0-58d816f553a1-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.391909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.391882 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.392006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.391927 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.392077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.392030 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.392077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.392054 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.392181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.392084 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.392181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.392162 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.392286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.392208 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.422121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.422097 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:45.494687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.494652 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.494832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.494704 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.494832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.494738 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.494832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.494765 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.494832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.494803 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.495098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.494843 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.495098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.494915 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.495098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.495020 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nvp\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-kube-api-access-f5nvp\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.495098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.495068 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26245fbe-359b-4ff6-b6b0-58d816f553a1-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.495664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.495636 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.495893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.495867 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26245fbe-359b-4ff6-b6b0-58d816f553a1-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.495991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.495906 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.496068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.495957 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.496298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.496273 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.500445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.498351 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.500445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.498761 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.549326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.549258 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.549674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.549655 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nvp\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-kube-api-access-f5nvp\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.627343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.627308 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:45.796268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.796229 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z"] Nov 02 21:29:45.801628 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:45.801571 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cb0869_4634_46af_b380_c1d04cae724a.slice/crio-fcd2a5d8cd4eb2a9f9e735fbb5d1f863cb771e00721f41820d36d819e181ec64 WatchSource:0}: Error finding container fcd2a5d8cd4eb2a9f9e735fbb5d1f863cb771e00721f41820d36d819e181ec64: Status 404 returned error can't find the container with id fcd2a5d8cd4eb2a9f9e735fbb5d1f863cb771e00721f41820d36d819e181ec64 Nov 02 21:29:45.974659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:45.974625 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" event={"ID":"b7cb0869-4634-46af-b380-c1d04cae724a","Type":"ContainerStarted","Data":"fcd2a5d8cd4eb2a9f9e735fbb5d1f863cb771e00721f41820d36d819e181ec64"} Nov 02 21:29:46.004419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.004385 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:46.012087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.012055 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg"] Nov 02 21:29:46.017533 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:46.017497 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26245fbe_359b_4ff6_b6b0_58d816f553a1.slice/crio-021437950f21b887f716bb3d272396cfc5b86e4d58b87f3f8fbc2a24e9e7971b WatchSource:0}: Error finding container 021437950f21b887f716bb3d272396cfc5b86e4d58b87f3f8fbc2a24e9e7971b: Status 404 returned error can't find the container with id 021437950f21b887f716bb3d272396cfc5b86e4d58b87f3f8fbc2a24e9e7971b Nov 02 21:29:46.020720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.020682 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:29:46.020831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.020767 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:29:46.020831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.020806 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:29:46.120922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.120895 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:46.178910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.178892 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:46.304950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.304912 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-openshift-ca\") pod \"224635dd-b127-4f40-a895-3cc532404a4f\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " Nov 02 21:29:46.305157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.305014 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224635dd-b127-4f40-a895-3cc532404a4f-kube-api-access-wbwkr\" (UniqueName: \"kubernetes.io/projected/224635dd-b127-4f40-a895-3cc532404a4f-kube-api-access-wbwkr\") pod \"224635dd-b127-4f40-a895-3cc532404a4f\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " Nov 02 21:29:46.305157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.305111 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-xks-ca\") pod \"224635dd-b127-4f40-a895-3cc532404a4f\" (UID: \"224635dd-b127-4f40-a895-3cc532404a4f\") " Nov 02 21:29:46.305670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.305306 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-openshift-ca" pod "224635dd-b127-4f40-a895-3cc532404a4f" (UID: "224635dd-b127-4f40-a895-3cc532404a4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:46.305670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.305633 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:46.307225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.307199 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224635dd-b127-4f40-a895-3cc532404a4f-kube-api-access-wbwkr" pod "224635dd-b127-4f40-a895-3cc532404a4f" (UID: "224635dd-b127-4f40-a895-3cc532404a4f"). InnerVolumeSpecName "kube-api-access-wbwkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:46.307596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.307576 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-xks-ca" pod "224635dd-b127-4f40-a895-3cc532404a4f" (UID: "224635dd-b127-4f40-a895-3cc532404a4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:46.406510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.406425 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbwkr\" (UniqueName: \"kubernetes.io/projected/224635dd-b127-4f40-a895-3cc532404a4f-kube-api-access-wbwkr\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:46.406510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.406451 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224635dd-b127-4f40-a895-3cc532404a4f-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:46.994019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.991403 4441 generic.go:356] "Generic (PLEG): container finished" podID="b7cb0869-4634-46af-b380-c1d04cae724a" containerID="fc3bb737321bf44e7c7d596b041008d26de340df307130e53feac836474cd30d" exitCode=0 Nov 02 21:29:46.994019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.991515 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" event={"ID":"b7cb0869-4634-46af-b380-c1d04cae724a","Type":"ContainerDied","Data":"fc3bb737321bf44e7c7d596b041008d26de340df307130e53feac836474cd30d"} Nov 02 21:29:46.994470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.994372 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" event={"ID":"224635dd-b127-4f40-a895-3cc532404a4f","Type":"ContainerDied","Data":"4abbdbe0eedd20cff6a09fccba1cca7f6f3351cbb1f5ec01260cef36b22b53c2"} Nov 02 21:29:46.994470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.994426 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4abbdbe0eedd20cff6a09fccba1cca7f6f3351cbb1f5ec01260cef36b22b53c2" Nov 02 21:29:46.994542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.994495 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv" Nov 02 21:29:46.998577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.998547 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" event={"ID":"26245fbe-359b-4ff6-b6b0-58d816f553a1","Type":"ContainerStarted","Data":"cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed"} Nov 02 21:29:46.998729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:46.998600 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" event={"ID":"26245fbe-359b-4ff6-b6b0-58d816f553a1","Type":"ContainerStarted","Data":"021437950f21b887f716bb3d272396cfc5b86e4d58b87f3f8fbc2a24e9e7971b"} Nov 02 21:29:47.147039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:47.147006 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:47.217989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:47.217294 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podStartSLOduration=2.217274066 podStartE2EDuration="2.217274066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:47.212560003 +0000 UTC m=+1016.768188339" watchObservedRunningTime="2026-11-02 21:29:47.217274066 +0000 UTC m=+1016.772902408" Nov 02 21:29:47.628037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:47.628002 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:47.629540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:47.629513 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:29:47.629672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:47.629558 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 21:29:47.720140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:47.720107 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:48.214436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.214400 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:48.326576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.326548 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7cb0869-4634-46af-b380-c1d04cae724a-kube-api-access-8nmtz\" (UniqueName: \"kubernetes.io/projected/b7cb0869-4634-46af-b380-c1d04cae724a-kube-api-access-8nmtz\") pod \"b7cb0869-4634-46af-b380-c1d04cae724a\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " Nov 02 21:29:48.326753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.326601 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-openshift-ca\") pod \"b7cb0869-4634-46af-b380-c1d04cae724a\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " Nov 02 21:29:48.326753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.326692 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-xks-ca\") pod \"b7cb0869-4634-46af-b380-c1d04cae724a\" (UID: \"b7cb0869-4634-46af-b380-c1d04cae724a\") " Nov 02 21:29:48.327271 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.327237 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-openshift-ca" pod "b7cb0869-4634-46af-b380-c1d04cae724a" (UID: "b7cb0869-4634-46af-b380-c1d04cae724a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:48.330196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.329213 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-xks-ca" pod "b7cb0869-4634-46af-b380-c1d04cae724a" (UID: "b7cb0869-4634-46af-b380-c1d04cae724a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:48.330196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.329849 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cb0869-4634-46af-b380-c1d04cae724a-kube-api-access-8nmtz" pod "b7cb0869-4634-46af-b380-c1d04cae724a" (UID: "b7cb0869-4634-46af-b380-c1d04cae724a"). InnerVolumeSpecName "kube-api-access-8nmtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:48.428461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.428426 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmtz\" (UniqueName: \"kubernetes.io/projected/b7cb0869-4634-46af-b380-c1d04cae724a-kube-api-access-8nmtz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:48.428461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.428461 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:48.428659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.428476 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7cb0869-4634-46af-b380-c1d04cae724a-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:48.628450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.628418 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:29:48.628611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.628483 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 21:29:48.832360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:48.832323 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:49.015226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.015135 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" event={"ID":"b7cb0869-4634-46af-b380-c1d04cae724a","Type":"ContainerDied","Data":"fcd2a5d8cd4eb2a9f9e735fbb5d1f863cb771e00721f41820d36d819e181ec64"} Nov 02 21:29:49.015226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.015196 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcd2a5d8cd4eb2a9f9e735fbb5d1f863cb771e00721f41820d36d819e181ec64" Nov 02 21:29:49.015226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.015212 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z" Nov 02 21:29:49.018345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.018313 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm"] Nov 02 21:29:49.019097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.019074 4441 cpu_manager.go:414] "containerMap: removing container" podUID="224635dd-b127-4f40-a895-3cc532404a4f" containerName="revoke-keys" Nov 02 21:29:49.019097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.019097 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224635dd-b127-4f40-a895-3cc532404a4f" containerName="revoke-keys" Nov 02 21:29:49.019224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.019109 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b7cb0869-4634-46af-b380-c1d04cae724a" containerName="revoke-keys" Nov 02 21:29:49.019224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.019118 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7cb0869-4634-46af-b380-c1d04cae724a" containerName="revoke-keys" Nov 02 21:29:49.019338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.019325 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="224635dd-b127-4f40-a895-3cc532404a4f" containerName="revoke-keys" Nov 02 21:29:49.019376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.019339 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7cb0869-4634-46af-b380-c1d04cae724a" containerName="revoke-keys" Nov 02 21:29:49.022949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.022925 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.101044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.094270 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm"] Nov 02 21:29:49.101538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.101513 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:29:49.115231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.115202 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:29:49.115326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.115252 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:29:49.115326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.115292 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:29:49.140371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.140332 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8x6w\" (UniqueName: \"kubernetes.io/projected/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-kube-api-access-c8x6w\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.140629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.140506 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.140629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.140603 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.241980 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.241922 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8x6w\" (UniqueName: \"kubernetes.io/projected/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-kube-api-access-c8x6w\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.242474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.242035 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.242474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.242095 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.242833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.242810 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.244582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.244556 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.300722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.300636 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8x6w\" (UniqueName: \"kubernetes.io/projected/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-kube-api-access-c8x6w\") pod \"maas-api-revoke-sub-50154a20-zgvrm\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.336816 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.336786 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:49.628326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.628289 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:29:49.628476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.628352 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 21:29:49.748476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.748440 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm"] Nov 02 21:29:49.753735 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:49.753707 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda671c0cc_37dd_4edb_bfb6_8d6228d3d1c0.slice/crio-02e2377761faefa03622d5c517e4613b34a830f8db7abb0f2aeb01bef906061e WatchSource:0}: Error finding container 02e2377761faefa03622d5c517e4613b34a830f8db7abb0f2aeb01bef906061e: Status 404 returned error can't find the container with id 02e2377761faefa03622d5c517e4613b34a830f8db7abb0f2aeb01bef906061e Nov 02 21:29:49.908908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.908820 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c"] Nov 02 21:29:49.914507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.914484 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:49.964143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:49.964120 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c"] Nov 02 21:29:50.037254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.037216 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" event={"ID":"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0","Type":"ContainerStarted","Data":"02e2377761faefa03622d5c517e4613b34a830f8db7abb0f2aeb01bef906061e"} Nov 02 21:29:50.050393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.050365 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.050556 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.050536 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4h28\" (UniqueName: \"kubernetes.io/projected/a88080f4-e327-4de3-bc87-47933fe0b2f9-kube-api-access-t4h28\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.050616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.050586 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.152270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.152235 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4h28\" (UniqueName: \"kubernetes.io/projected/a88080f4-e327-4de3-bc87-47933fe0b2f9-kube-api-access-t4h28\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.152466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.152288 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.152466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.152329 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.153113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.153086 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.154576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.154555 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.206478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.206450 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4h28\" (UniqueName: \"kubernetes.io/projected/a88080f4-e327-4de3-bc87-47933fe0b2f9-kube-api-access-t4h28\") pod \"maas-api-revoke-sub-03d77d8a-9jt2c\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.230005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.229958 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:50.589226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.588532 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:50.628737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.628705 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:29:50.628894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.628749 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 21:29:50.644301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:50.644270 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c"] Nov 02 21:29:50.651836 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:50.651186 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88080f4_e327_4de3_bc87_47933fe0b2f9.slice/crio-7becbb28ae82434b2e17891d6e6720bb09ba9a338c41382dc4f5376962e960e1 WatchSource:0}: Error finding container 7becbb28ae82434b2e17891d6e6720bb09ba9a338c41382dc4f5376962e960e1: Status 404 returned error can't find the container with id 7becbb28ae82434b2e17891d6e6720bb09ba9a338c41382dc4f5376962e960e1 Nov 02 21:29:51.045144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:51.045104 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" event={"ID":"a88080f4-e327-4de3-bc87-47933fe0b2f9","Type":"ContainerStarted","Data":"7becbb28ae82434b2e17891d6e6720bb09ba9a338c41382dc4f5376962e960e1"} Nov 02 21:29:51.046628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:51.046601 4441 generic.go:356] "Generic (PLEG): container finished" podID="a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" containerID="dadcda358b286a08b8cd5f5c6142bb6a1a8f55cef50b0e6411824fa534b03620" exitCode=0 Nov 02 21:29:51.046740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:51.046657 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" event={"ID":"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0","Type":"ContainerDied","Data":"dadcda358b286a08b8cd5f5c6142bb6a1a8f55cef50b0e6411824fa534b03620"} Nov 02 21:29:51.628423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:51.628391 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:29:51.628792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:51.628441 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 21:29:52.055249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.055158 4441 generic.go:356] "Generic (PLEG): container finished" podID="a88080f4-e327-4de3-bc87-47933fe0b2f9" containerID="600ecfa5a9ed5b2860ff7ea624baaa35039ebbb5de317eb7085d34914d3ed2a8" exitCode=0 Nov 02 21:29:52.055249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.055208 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" event={"ID":"a88080f4-e327-4de3-bc87-47933fe0b2f9","Type":"ContainerDied","Data":"600ecfa5a9ed5b2860ff7ea624baaa35039ebbb5de317eb7085d34914d3ed2a8"} Nov 02 21:29:52.221286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.221263 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:52.416310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.416271 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-kube-api-access-c8x6w\" (UniqueName: \"kubernetes.io/projected/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-kube-api-access-c8x6w\") pod \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " Nov 02 21:29:52.416483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.416363 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-openshift-ca\") pod \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " Nov 02 21:29:52.416483 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.416436 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-xks-ca\") pod \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\" (UID: \"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0\") " Nov 02 21:29:52.417320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.417272 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-openshift-ca" pod "a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" (UID: "a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:52.418775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.418740 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-kube-api-access-c8x6w" pod "a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" (UID: "a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0"). InnerVolumeSpecName "kube-api-access-c8x6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:52.419013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.418990 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-xks-ca" pod "a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" (UID: "a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:52.518225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.518195 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:52.518225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.518225 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8x6w\" (UniqueName: \"kubernetes.io/projected/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-kube-api-access-c8x6w\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:52.518393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.518241 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:52.872058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.872018 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:52.873285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.873258 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:52.875073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:52.875015 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:29:53.065408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.065376 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" Nov 02 21:29:53.065589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.065409 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm" event={"ID":"a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0","Type":"ContainerDied","Data":"02e2377761faefa03622d5c517e4613b34a830f8db7abb0f2aeb01bef906061e"} Nov 02 21:29:53.065589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.065468 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02e2377761faefa03622d5c517e4613b34a830f8db7abb0f2aeb01bef906061e" Nov 02 21:29:53.279751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.279722 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:53.327823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.327780 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88080f4-e327-4de3-bc87-47933fe0b2f9-kube-api-access-t4h28\" (UniqueName: \"kubernetes.io/projected/a88080f4-e327-4de3-bc87-47933fe0b2f9-kube-api-access-t4h28\") pod \"a88080f4-e327-4de3-bc87-47933fe0b2f9\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " Nov 02 21:29:53.328001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.327906 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-xks-ca\") pod \"a88080f4-e327-4de3-bc87-47933fe0b2f9\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " Nov 02 21:29:53.328001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.327964 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-openshift-ca\") pod \"a88080f4-e327-4de3-bc87-47933fe0b2f9\" (UID: \"a88080f4-e327-4de3-bc87-47933fe0b2f9\") " Nov 02 21:29:53.328348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.328316 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-openshift-ca" pod "a88080f4-e327-4de3-bc87-47933fe0b2f9" (UID: "a88080f4-e327-4de3-bc87-47933fe0b2f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:53.328513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.328488 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:53.330467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.330441 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-xks-ca" pod "a88080f4-e327-4de3-bc87-47933fe0b2f9" (UID: "a88080f4-e327-4de3-bc87-47933fe0b2f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:53.330757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.330737 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88080f4-e327-4de3-bc87-47933fe0b2f9-kube-api-access-t4h28" pod "a88080f4-e327-4de3-bc87-47933fe0b2f9" (UID: "a88080f4-e327-4de3-bc87-47933fe0b2f9"). InnerVolumeSpecName "kube-api-access-t4h28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:53.429147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.429072 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4h28\" (UniqueName: \"kubernetes.io/projected/a88080f4-e327-4de3-bc87-47933fe0b2f9-kube-api-access-t4h28\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:53.429147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:53.429102 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88080f4-e327-4de3-bc87-47933fe0b2f9-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:54.095392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:54.095359 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" Nov 02 21:29:54.100077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:54.100046 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c" event={"ID":"a88080f4-e327-4de3-bc87-47933fe0b2f9","Type":"ContainerDied","Data":"7becbb28ae82434b2e17891d6e6720bb09ba9a338c41382dc4f5376962e960e1"} Nov 02 21:29:54.100282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:54.100264 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7becbb28ae82434b2e17891d6e6720bb09ba9a338c41382dc4f5376962e960e1" Nov 02 21:29:54.100832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:54.100812 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:54.138695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:54.138666 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:55.108772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.108743 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:29:55.189636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.189596 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:55.850513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.850477 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn"] Nov 02 21:29:55.851042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.851023 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a88080f4-e327-4de3-bc87-47933fe0b2f9" containerName="revoke-keys" Nov 02 21:29:55.851042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.851041 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88080f4-e327-4de3-bc87-47933fe0b2f9" containerName="revoke-keys" Nov 02 21:29:55.851189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.851061 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" containerName="revoke-keys" Nov 02 21:29:55.851189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.851067 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" containerName="revoke-keys" Nov 02 21:29:55.851189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.851157 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88080f4-e327-4de3-bc87-47933fe0b2f9" containerName="revoke-keys" Nov 02 21:29:55.851189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.851168 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" containerName="revoke-keys" Nov 02 21:29:55.854517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.854501 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:55.909445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.909414 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn"] Nov 02 21:29:55.938930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.938891 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:55.939073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.939042 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:55.939133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.939118 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6lh\" (UniqueName: \"kubernetes.io/projected/45e66e75-8f28-4c0c-8a38-843bfebcca69-kube-api-access-td6lh\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:55.949909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.949875 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:29:55.949909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.949886 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:29:55.949909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.949886 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:29:55.951558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:55.951542 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:29:56.039810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.039775 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:56.040008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.039828 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:56.040008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.039873 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6lh\" (UniqueName: \"kubernetes.io/projected/45e66e75-8f28-4c0c-8a38-843bfebcca69-kube-api-access-td6lh\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:56.040436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.040414 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:56.042294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.042270 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:56.094605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.094575 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6lh\" (UniqueName: \"kubernetes.io/projected/45e66e75-8f28-4c0c-8a38-843bfebcca69-kube-api-access-td6lh\") pod \"maas-api-revoke-sub-7038caa5-qc2wn\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:56.167760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.167695 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:56.545685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:56.545600 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn"] Nov 02 21:29:56.551350 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:56.551327 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e66e75_8f28_4c0c_8a38_843bfebcca69.slice/crio-3cbf25b4ab20ae038da8e47c88daff730f99d880bea3b6d97d8078d6da1b9d81 WatchSource:0}: Error finding container 3cbf25b4ab20ae038da8e47c88daff730f99d880bea3b6d97d8078d6da1b9d81: Status 404 returned error can't find the container with id 3cbf25b4ab20ae038da8e47c88daff730f99d880bea3b6d97d8078d6da1b9d81 Nov 02 21:29:57.008024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.007989 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv"] Nov 02 21:29:57.012759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.012737 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.066696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.066667 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv"] Nov 02 21:29:57.125888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.125857 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" event={"ID":"45e66e75-8f28-4c0c-8a38-843bfebcca69","Type":"ContainerStarted","Data":"f1caa4aa32d021a121d016c8b0180ba1d19eebc1083c3e7a72d92bef7794253f"} Nov 02 21:29:57.126064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.125908 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" event={"ID":"45e66e75-8f28-4c0c-8a38-843bfebcca69","Type":"ContainerStarted","Data":"3cbf25b4ab20ae038da8e47c88daff730f99d880bea3b6d97d8078d6da1b9d81"} Nov 02 21:29:57.152889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.152820 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hs2\" (UniqueName: \"kubernetes.io/projected/d34eee63-3b3b-4b59-b525-6eee4e98a05b-kube-api-access-l6hs2\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.152889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.152855 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.153044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.153014 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.236948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.236896 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" podStartSLOduration=2.236883461 podStartE2EDuration="2.236883461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:57.232193415 +0000 UTC m=+1026.787821755" watchObservedRunningTime="2026-11-02 21:29:57.236883461 +0000 UTC m=+1026.792511797" Nov 02 21:29:57.253623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.253588 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hs2\" (UniqueName: \"kubernetes.io/projected/d34eee63-3b3b-4b59-b525-6eee4e98a05b-kube-api-access-l6hs2\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.253745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.253630 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.253803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.253786 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.254437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.254409 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.255948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.255929 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.307273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.307241 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hs2\" (UniqueName: \"kubernetes.io/projected/d34eee63-3b3b-4b59-b525-6eee4e98a05b-kube-api-access-l6hs2\") pod \"maas-api-revoke-sub-816742d1-l4jwv\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.326709 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.326683 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:29:57.710782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:57.710631 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv"] Nov 02 21:29:57.713285 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:29:57.713259 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34eee63_3b3b_4b59_b525_6eee4e98a05b.slice/crio-494c5288766e1e14edb9e811471cd6d992ec96fb0b35dc4e24a2579f46141580 WatchSource:0}: Error finding container 494c5288766e1e14edb9e811471cd6d992ec96fb0b35dc4e24a2579f46141580: Status 404 returned error can't find the container with id 494c5288766e1e14edb9e811471cd6d992ec96fb0b35dc4e24a2579f46141580 Nov 02 21:29:58.134662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:58.134624 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" event={"ID":"d34eee63-3b3b-4b59-b525-6eee4e98a05b","Type":"ContainerStarted","Data":"fecdcac34ac6ba2f9887a0b1fa4ff214848b2efe93a77c8e14a27f5c26b641cb"} Nov 02 21:29:58.134826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:58.134676 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" event={"ID":"d34eee63-3b3b-4b59-b525-6eee4e98a05b","Type":"ContainerStarted","Data":"494c5288766e1e14edb9e811471cd6d992ec96fb0b35dc4e24a2579f46141580"} Nov 02 21:29:58.136356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:58.136330 4441 generic.go:356] "Generic (PLEG): container finished" podID="45e66e75-8f28-4c0c-8a38-843bfebcca69" containerID="f1caa4aa32d021a121d016c8b0180ba1d19eebc1083c3e7a72d92bef7794253f" exitCode=0 Nov 02 21:29:58.136470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:58.136411 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" event={"ID":"45e66e75-8f28-4c0c-8a38-843bfebcca69","Type":"ContainerDied","Data":"f1caa4aa32d021a121d016c8b0180ba1d19eebc1083c3e7a72d92bef7794253f"} Nov 02 21:29:58.246470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:58.246367 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" podStartSLOduration=2.24635468 podStartE2EDuration="2.24635468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:58.244344223 +0000 UTC m=+1027.799972572" watchObservedRunningTime="2026-11-02 21:29:58.24635468 +0000 UTC m=+1027.801983017" Nov 02 21:29:59.147815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.147776 4441 generic.go:356] "Generic (PLEG): container finished" podID="d34eee63-3b3b-4b59-b525-6eee4e98a05b" containerID="fecdcac34ac6ba2f9887a0b1fa4ff214848b2efe93a77c8e14a27f5c26b641cb" exitCode=0 Nov 02 21:29:59.148008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.147864 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" event={"ID":"d34eee63-3b3b-4b59-b525-6eee4e98a05b","Type":"ContainerDied","Data":"fecdcac34ac6ba2f9887a0b1fa4ff214848b2efe93a77c8e14a27f5c26b641cb"} Nov 02 21:29:59.308026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.307996 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:29:59.377726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.377697 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-openshift-ca\") pod \"45e66e75-8f28-4c0c-8a38-843bfebcca69\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " Nov 02 21:29:59.377896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.377790 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e66e75-8f28-4c0c-8a38-843bfebcca69-kube-api-access-td6lh\" (UniqueName: \"kubernetes.io/projected/45e66e75-8f28-4c0c-8a38-843bfebcca69-kube-api-access-td6lh\") pod \"45e66e75-8f28-4c0c-8a38-843bfebcca69\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " Nov 02 21:29:59.377896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.377825 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-xks-ca\") pod \"45e66e75-8f28-4c0c-8a38-843bfebcca69\" (UID: \"45e66e75-8f28-4c0c-8a38-843bfebcca69\") " Nov 02 21:29:59.378153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.378119 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-openshift-ca" pod "45e66e75-8f28-4c0c-8a38-843bfebcca69" (UID: "45e66e75-8f28-4c0c-8a38-843bfebcca69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:29:59.379784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.379755 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-xks-ca" pod "45e66e75-8f28-4c0c-8a38-843bfebcca69" (UID: "45e66e75-8f28-4c0c-8a38-843bfebcca69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:29:59.379904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.379887 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e66e75-8f28-4c0c-8a38-843bfebcca69-kube-api-access-td6lh" pod "45e66e75-8f28-4c0c-8a38-843bfebcca69" (UID: "45e66e75-8f28-4c0c-8a38-843bfebcca69"). InnerVolumeSpecName "kube-api-access-td6lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:29:59.479154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.479076 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:59.479154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.479109 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6lh\" (UniqueName: \"kubernetes.io/projected/45e66e75-8f28-4c0c-8a38-843bfebcca69-kube-api-access-td6lh\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:59.479154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.479120 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e66e75-8f28-4c0c-8a38-843bfebcca69-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:29:59.654548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:29:59.654518 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:00.156830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.156798 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" Nov 02 21:30:00.156830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.156815 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn" event={"ID":"45e66e75-8f28-4c0c-8a38-843bfebcca69","Type":"ContainerDied","Data":"3cbf25b4ab20ae038da8e47c88daff730f99d880bea3b6d97d8078d6da1b9d81"} Nov 02 21:30:00.157024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.156859 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cbf25b4ab20ae038da8e47c88daff730f99d880bea3b6d97d8078d6da1b9d81" Nov 02 21:30:00.163160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.163133 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj"] Nov 02 21:30:00.163845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.163823 4441 cpu_manager.go:414] "containerMap: removing container" podUID="45e66e75-8f28-4c0c-8a38-843bfebcca69" containerName="revoke-keys" Nov 02 21:30:00.163932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.163847 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e66e75-8f28-4c0c-8a38-843bfebcca69" containerName="revoke-keys" Nov 02 21:30:00.164035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.164023 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e66e75-8f28-4c0c-8a38-843bfebcca69" containerName="revoke-keys" Nov 02 21:30:00.168079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.168060 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.169858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.169843 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt"] Nov 02 21:30:00.173563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.173544 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.174770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.174752 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g"] Nov 02 21:30:00.179055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.179028 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.179665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.179647 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7"] Nov 02 21:30:00.183382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.183357 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.188316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.188291 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9"] Nov 02 21:30:00.192290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.192275 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.271105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.271071 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj"] Nov 02 21:30:00.281936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.281902 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g"] Nov 02 21:30:00.286406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.286377 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt"] Nov 02 21:30:00.288259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.288225 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941bf58c-cda8-4103-9b66-9d45b0476bbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-7zh5g\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.288456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.288436 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27961744-e82b-4d10-9fae-5206f8a0d227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.288747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.288726 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.288924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.288909 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.294638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.289077 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwf8\" (UniqueName: \"kubernetes.io/projected/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-kube-api-access-2hwf8\") pod \"maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.294638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.293081 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klv9r\" (UniqueName: \"kubernetes.io/projected/27961744-e82b-4d10-9fae-5206f8a0d227-kube-api-access-klv9r\") pod \"maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.294638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.293118 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df26296c-411c-4a5e-8261-d7e03612d9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.294638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.293761 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbf8d\" (UniqueName: \"kubernetes.io/projected/df26296c-411c-4a5e-8261-d7e03612d9e3-kube-api-access-wbf8d\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.294638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.293808 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb27\" (UniqueName: \"kubernetes.io/projected/941bf58c-cda8-4103-9b66-9d45b0476bbe-kube-api-access-kwb27\") pod \"maas-api-key-cleanup-29894250-7zh5g\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.294638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.293885 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gq9d\" (UniqueName: \"kubernetes.io/projected/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-kube-api-access-4gq9d\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.305479 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.305453 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9"] Nov 02 21:30:00.309806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.309696 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7"] Nov 02 21:30:00.349458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.349429 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:30:00.396006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.395954 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27961744-e82b-4d10-9fae-5206f8a0d227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.396147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396070 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.396147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396098 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.396147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwf8\" (UniqueName: \"kubernetes.io/projected/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-kube-api-access-2hwf8\") pod \"maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.396147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396142 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klv9r\" (UniqueName: \"kubernetes.io/projected/27961744-e82b-4d10-9fae-5206f8a0d227-kube-api-access-klv9r\") pod \"maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.396368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396291 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df26296c-411c-4a5e-8261-d7e03612d9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.396426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396356 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbf8d\" (UniqueName: \"kubernetes.io/projected/df26296c-411c-4a5e-8261-d7e03612d9e3-kube-api-access-wbf8d\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.396426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396415 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb27\" (UniqueName: \"kubernetes.io/projected/941bf58c-cda8-4103-9b66-9d45b0476bbe-kube-api-access-kwb27\") pod \"maas-api-key-cleanup-29894250-7zh5g\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.396539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396465 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gq9d\" (UniqueName: \"kubernetes.io/projected/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-kube-api-access-4gq9d\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.396602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396555 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941bf58c-cda8-4103-9b66-9d45b0476bbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-7zh5g\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.396825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396805 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27961744-e82b-4d10-9fae-5206f8a0d227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.396996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.396888 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.397094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.397065 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df26296c-411c-4a5e-8261-d7e03612d9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.397308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.397290 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941bf58c-cda8-4103-9b66-9d45b0476bbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-7zh5g\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.397373 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.397327 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.450724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.450649 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwf8\" (UniqueName: \"kubernetes.io/projected/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-kube-api-access-2hwf8\") pod \"maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.450724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.450658 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klv9r\" (UniqueName: \"kubernetes.io/projected/27961744-e82b-4d10-9fae-5206f8a0d227-kube-api-access-klv9r\") pod \"maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.450724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.450698 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbf8d\" (UniqueName: \"kubernetes.io/projected/df26296c-411c-4a5e-8261-d7e03612d9e3-kube-api-access-wbf8d\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.451024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.450864 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gq9d\" (UniqueName: \"kubernetes.io/projected/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-kube-api-access-4gq9d\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.451195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.451176 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb27\" (UniqueName: \"kubernetes.io/projected/941bf58c-cda8-4103-9b66-9d45b0476bbe-kube-api-access-kwb27\") pod \"maas-api-key-cleanup-29894250-7zh5g\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.485345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.485325 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:00.496648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.496625 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:00.497932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.497912 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34eee63-3b3b-4b59-b525-6eee4e98a05b-kube-api-access-l6hs2\" (UniqueName: \"kubernetes.io/projected/d34eee63-3b3b-4b59-b525-6eee4e98a05b-kube-api-access-l6hs2\") pod \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " Nov 02 21:30:00.498087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.497962 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-xks-ca\") pod \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " Nov 02 21:30:00.498087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.498031 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-openshift-ca\") pod \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\" (UID: \"d34eee63-3b3b-4b59-b525-6eee4e98a05b\") " Nov 02 21:30:00.498453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.498430 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-openshift-ca" pod "d34eee63-3b3b-4b59-b525-6eee4e98a05b" (UID: "d34eee63-3b3b-4b59-b525-6eee4e98a05b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:00.501097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.501077 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-xks-ca" pod "d34eee63-3b3b-4b59-b525-6eee4e98a05b" (UID: "d34eee63-3b3b-4b59-b525-6eee4e98a05b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:30:00.501253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.501116 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34eee63-3b3b-4b59-b525-6eee4e98a05b-kube-api-access-l6hs2" pod "d34eee63-3b3b-4b59-b525-6eee4e98a05b" (UID: "d34eee63-3b3b-4b59-b525-6eee4e98a05b"). InnerVolumeSpecName "kube-api-access-l6hs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:00.504475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.504454 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:00.515386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.515362 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:00.522674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.522650 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:00.604686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.604652 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hs2\" (UniqueName: \"kubernetes.io/projected/d34eee63-3b3b-4b59-b525-6eee4e98a05b-kube-api-access-l6hs2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:00.604804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.604688 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:00.604804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.604704 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d34eee63-3b3b-4b59-b525-6eee4e98a05b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:00.878065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.878030 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj"] Nov 02 21:30:00.883106 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:00.883074 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bc0aeb_c8ca_4ed4_afe4_217c6085c1b9.slice/crio-de0e28f69da1ed636b13532daf49417851c82ace5a668c04396e59428aeccacf WatchSource:0}: Error finding container de0e28f69da1ed636b13532daf49417851c82ace5a668c04396e59428aeccacf: Status 404 returned error can't find the container with id de0e28f69da1ed636b13532daf49417851c82ace5a668c04396e59428aeccacf Nov 02 21:30:00.929681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.929654 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7"] Nov 02 21:30:00.933540 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:00.933515 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf26296c_411c_4a5e_8261_d7e03612d9e3.slice/crio-aef6c5988cf3487826ed716bfe2f2664d681ee55179fca3bf3e1e746b39a02b2 WatchSource:0}: Error finding container aef6c5988cf3487826ed716bfe2f2664d681ee55179fca3bf3e1e746b39a02b2: Status 404 returned error can't find the container with id aef6c5988cf3487826ed716bfe2f2664d681ee55179fca3bf3e1e746b39a02b2 Nov 02 21:30:00.934142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.934121 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g"] Nov 02 21:30:00.936718 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:00.936482 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941bf58c_cda8_4103_9b66_9d45b0476bbe.slice/crio-edae59eb72d84b49eb41f2ff1fc06d5493a98a00c204060a6b745bddc0212a37 WatchSource:0}: Error finding container edae59eb72d84b49eb41f2ff1fc06d5493a98a00c204060a6b745bddc0212a37: Status 404 returned error can't find the container with id edae59eb72d84b49eb41f2ff1fc06d5493a98a00c204060a6b745bddc0212a37 Nov 02 21:30:00.978789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.978765 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9"] Nov 02 21:30:00.982315 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:00.982291 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27961744_e82b_4d10_9fae_5206f8a0d227.slice/crio-66058beef3146fc27d37244fb4b02716146be8ce169bfb525c2c05f094bab5e4 WatchSource:0}: Error finding container 66058beef3146fc27d37244fb4b02716146be8ce169bfb525c2c05f094bab5e4: Status 404 returned error can't find the container with id 66058beef3146fc27d37244fb4b02716146be8ce169bfb525c2c05f094bab5e4 Nov 02 21:30:00.983383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:00.983362 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt"] Nov 02 21:30:00.987096 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:00.987072 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4994be9a_363f_4f43_a0d4_b9e9bc8121d8.slice/crio-15e529f05be6b0a413c2c1ef9da00d955a671a3cbb1aeb8108b976091cfa3032 WatchSource:0}: Error finding container 15e529f05be6b0a413c2c1ef9da00d955a671a3cbb1aeb8108b976091cfa3032: Status 404 returned error can't find the container with id 15e529f05be6b0a413c2c1ef9da00d955a671a3cbb1aeb8108b976091cfa3032 Nov 02 21:30:01.165139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.165044 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" event={"ID":"df26296c-411c-4a5e-8261-d7e03612d9e3","Type":"ContainerStarted","Data":"aef6c5988cf3487826ed716bfe2f2664d681ee55179fca3bf3e1e746b39a02b2"} Nov 02 21:30:01.166353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.166324 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" event={"ID":"941bf58c-cda8-4103-9b66-9d45b0476bbe","Type":"ContainerStarted","Data":"edae59eb72d84b49eb41f2ff1fc06d5493a98a00c204060a6b745bddc0212a37"} Nov 02 21:30:01.167798 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.167773 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" event={"ID":"d34eee63-3b3b-4b59-b525-6eee4e98a05b","Type":"ContainerDied","Data":"494c5288766e1e14edb9e811471cd6d992ec96fb0b35dc4e24a2579f46141580"} Nov 02 21:30:01.167889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.167820 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="494c5288766e1e14edb9e811471cd6d992ec96fb0b35dc4e24a2579f46141580" Nov 02 21:30:01.167889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.167790 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv" Nov 02 21:30:01.169312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.169131 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" event={"ID":"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9","Type":"ContainerStarted","Data":"de0e28f69da1ed636b13532daf49417851c82ace5a668c04396e59428aeccacf"} Nov 02 21:30:01.171642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.171216 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" event={"ID":"27961744-e82b-4d10-9fae-5206f8a0d227","Type":"ContainerStarted","Data":"66058beef3146fc27d37244fb4b02716146be8ce169bfb525c2c05f094bab5e4"} Nov 02 21:30:01.173636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.173202 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" event={"ID":"4994be9a-363f-4f43-a0d4-b9e9bc8121d8","Type":"ContainerStarted","Data":"15e529f05be6b0a413c2c1ef9da00d955a671a3cbb1aeb8108b976091cfa3032"} Nov 02 21:30:01.584169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.584134 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw"] Nov 02 21:30:01.584713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.584697 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d34eee63-3b3b-4b59-b525-6eee4e98a05b" containerName="revoke-keys" Nov 02 21:30:01.584766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.584714 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34eee63-3b3b-4b59-b525-6eee4e98a05b" containerName="revoke-keys" Nov 02 21:30:01.584819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.584810 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34eee63-3b3b-4b59-b525-6eee4e98a05b" containerName="revoke-keys" Nov 02 21:30:01.588382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.588366 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.638216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.638191 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-284b8e-openshift-default-dockercfg-w2r92\"" Nov 02 21:30:01.644685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.644661 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw"] Nov 02 21:30:01.716056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716022 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716081 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9492a3-2097-4035-b212-ae86afc0667b-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716192 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtk7\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-kube-api-access-nbtk7\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716224 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716262 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716326 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716413 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716429 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9492a3-2097-4035-b212-ae86afc0667b-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.716567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.716448 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.817800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.817771 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.817903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.817803 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9492a3-2097-4035-b212-ae86afc0667b-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.817903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.817826 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.817903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.817877 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.817903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.817898 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9492a3-2097-4035-b212-ae86afc0667b-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.817927 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtk7\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-kube-api-access-nbtk7\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.817953 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.818052 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.818746 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.818358 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.818670 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.818829 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9492a3-2097-4035-b212-ae86afc0667b-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.818625 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.818998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.818852 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.820646 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.820628 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9492a3-2097-4035-b212-ae86afc0667b-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.820714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.820632 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.872506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.872347 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.872642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.872598 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtk7\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-kube-api-access-nbtk7\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:01.901888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:01.901866 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:02.183259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.183223 4441 generic.go:356] "Generic (PLEG): container finished" podID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerID="430c90fb74e7864d9d4d416df201c027cce1cd580eeceab4d50cc215a3fc985d" exitCode=60 Nov 02 21:30:02.183434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.183306 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" event={"ID":"df26296c-411c-4a5e-8261-d7e03612d9e3","Type":"ContainerDied","Data":"430c90fb74e7864d9d4d416df201c027cce1cd580eeceab4d50cc215a3fc985d"} Nov 02 21:30:02.183612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.183592 4441 scope.go:122] "RemoveContainer" containerID="430c90fb74e7864d9d4d416df201c027cce1cd580eeceab4d50cc215a3fc985d" Nov 02 21:30:02.185174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.185058 4441 generic.go:356] "Generic (PLEG): container finished" podID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerID="1b7a042c16981b069c0dfc4a27213ab5c5ce91ee1e79b03c3438b51b5913df71" exitCode=60 Nov 02 21:30:02.185174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.185087 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" event={"ID":"941bf58c-cda8-4103-9b66-9d45b0476bbe","Type":"ContainerDied","Data":"1b7a042c16981b069c0dfc4a27213ab5c5ce91ee1e79b03c3438b51b5913df71"} Nov 02 21:30:02.185382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.185360 4441 scope.go:122] "RemoveContainer" containerID="1b7a042c16981b069c0dfc4a27213ab5c5ce91ee1e79b03c3438b51b5913df71" Nov 02 21:30:02.187029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.186886 4441 generic.go:356] "Generic (PLEG): container finished" podID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerID="4347eb590ff1a79e29ce29188f3102f802466371f7d8c74955359134bf144696" exitCode=60 Nov 02 21:30:02.187029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.186992 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" event={"ID":"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9","Type":"ContainerDied","Data":"4347eb590ff1a79e29ce29188f3102f802466371f7d8c74955359134bf144696"} Nov 02 21:30:02.187239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.187212 4441 scope.go:122] "RemoveContainer" containerID="4347eb590ff1a79e29ce29188f3102f802466371f7d8c74955359134bf144696" Nov 02 21:30:02.188633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.188613 4441 generic.go:356] "Generic (PLEG): container finished" podID="27961744-e82b-4d10-9fae-5206f8a0d227" containerID="23f1dc6b2f39afc4394311b4c6eb6998439291733a590da2b1379950e2b74073" exitCode=60 Nov 02 21:30:02.188717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.188675 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" event={"ID":"27961744-e82b-4d10-9fae-5206f8a0d227","Type":"ContainerDied","Data":"23f1dc6b2f39afc4394311b4c6eb6998439291733a590da2b1379950e2b74073"} Nov 02 21:30:02.188882 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.188866 4441 scope.go:122] "RemoveContainer" containerID="23f1dc6b2f39afc4394311b4c6eb6998439291733a590da2b1379950e2b74073" Nov 02 21:30:02.190937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.190913 4441 generic.go:356] "Generic (PLEG): container finished" podID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerID="b9a7f7418b5663ff5b0f7581bf899198ee252705505983c32c0d6f6764d22226" exitCode=60 Nov 02 21:30:02.191064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.191003 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" event={"ID":"4994be9a-363f-4f43-a0d4-b9e9bc8121d8","Type":"ContainerDied","Data":"b9a7f7418b5663ff5b0f7581bf899198ee252705505983c32c0d6f6764d22226"} Nov 02 21:30:02.191214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.191198 4441 scope.go:122] "RemoveContainer" containerID="b9a7f7418b5663ff5b0f7581bf899198ee252705505983c32c0d6f6764d22226" Nov 02 21:30:02.353098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.352724 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw"] Nov 02 21:30:02.374431 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:02.374399 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9492a3_2097_4035_b212_ae86afc0667b.slice/crio-d102bddf5d8cc15288d279aa879cbd6a45893215dd74c0a9444de22306a8a694 WatchSource:0}: Error finding container d102bddf5d8cc15288d279aa879cbd6a45893215dd74c0a9444de22306a8a694: Status 404 returned error can't find the container with id d102bddf5d8cc15288d279aa879cbd6a45893215dd74c0a9444de22306a8a694 Nov 02 21:30:02.387922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.387227 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:30:02.387922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.387392 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:30:02.387922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:02.387485 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:30:03.208661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.208620 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" event={"ID":"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9","Type":"ContainerStarted","Data":"ba85618a5fd19a03858e4b9ccb48174a3c696724d4229f959868bb257e027e67"} Nov 02 21:30:03.210641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.210615 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" event={"ID":"27961744-e82b-4d10-9fae-5206f8a0d227","Type":"ContainerStarted","Data":"d2ef1a0b8a652ea53b8e897258bc5f011e46a8214c1b27e16aef5a4f4e7bd35d"} Nov 02 21:30:03.212747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.212722 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" event={"ID":"4994be9a-363f-4f43-a0d4-b9e9bc8121d8","Type":"ContainerStarted","Data":"ded85024047be832880922c6258bbff089333c5a895641dca8f6ffdd229f8f1c"} Nov 02 21:30:03.214512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.214490 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" event={"ID":"da9492a3-2097-4035-b212-ae86afc0667b","Type":"ContainerStarted","Data":"df7b22466548e2dea8b74c1f9f707a5440ebc7a41aa94fc9b53c31822a4c26d7"} Nov 02 21:30:03.214613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.214534 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" event={"ID":"da9492a3-2097-4035-b212-ae86afc0667b","Type":"ContainerStarted","Data":"d102bddf5d8cc15288d279aa879cbd6a45893215dd74c0a9444de22306a8a694"} Nov 02 21:30:03.216357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.216336 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" event={"ID":"df26296c-411c-4a5e-8261-d7e03612d9e3","Type":"ContainerStarted","Data":"967e5369adf7ce13fa6233a1e83a3bca6da9d8068e1deb96767fbc9f7c493523"} Nov 02 21:30:03.218128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.218109 4441 generic.go:356] "Generic (PLEG): container finished" podID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerID="4d9604f804176a44388f3a844e8876abbb61d0e118b9092297319396bb6d31ab" exitCode=60 Nov 02 21:30:03.218210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.218188 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" event={"ID":"941bf58c-cda8-4103-9b66-9d45b0476bbe","Type":"ContainerDied","Data":"4d9604f804176a44388f3a844e8876abbb61d0e118b9092297319396bb6d31ab"} Nov 02 21:30:03.218259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.218245 4441 scope.go:122] "RemoveContainer" containerID="1b7a042c16981b069c0dfc4a27213ab5c5ce91ee1e79b03c3438b51b5913df71" Nov 02 21:30:03.218341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.218327 4441 scope.go:122] "RemoveContainer" containerID="4d9604f804176a44388f3a844e8876abbb61d0e118b9092297319396bb6d31ab" Nov 02 21:30:03.218574 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:03.218550 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-7zh5g_odh-ai-gateway-infra(941bf58c-cda8-4103-9b66-9d45b0476bbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" Nov 02 21:30:03.318500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.318450 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" podStartSLOduration=3.318437591 podStartE2EDuration="3.318437591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:03.314694338 +0000 UTC m=+1032.870322678" watchObservedRunningTime="2026-11-02 21:30:03.318437591 +0000 UTC m=+1032.874065927" Nov 02 21:30:03.426896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.426842 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" podStartSLOduration=3.426827927 podStartE2EDuration="3.426827927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:03.422433906 +0000 UTC m=+1032.978062254" watchObservedRunningTime="2026-11-02 21:30:03.426827927 +0000 UTC m=+1032.982456265" Nov 02 21:30:03.556436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.556375 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" podStartSLOduration=3.55634491 podStartE2EDuration="3.55634491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:03.552281466 +0000 UTC m=+1033.107909836" watchObservedRunningTime="2026-11-02 21:30:03.55634491 +0000 UTC m=+1033.111973248" Nov 02 21:30:03.790006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.789858 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podStartSLOduration=2.789834678 podStartE2EDuration="2.789834678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:03.767860381 +0000 UTC m=+1033.323488742" watchObservedRunningTime="2026-11-02 21:30:03.789834678 +0000 UTC m=+1033.345463016" Nov 02 21:30:03.892993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.892308 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" podStartSLOduration=3.892289985 podStartE2EDuration="3.892289985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:03.891791891 +0000 UTC m=+1033.447420229" watchObservedRunningTime="2026-11-02 21:30:03.892289985 +0000 UTC m=+1033.447918323" Nov 02 21:30:03.902296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.902264 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:03.904107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.904077 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:30:03.904222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:03.904133 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 21:30:04.192991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.192728 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:04.228026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.227993 4441 generic.go:356] "Generic (PLEG): container finished" podID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerID="967e5369adf7ce13fa6233a1e83a3bca6da9d8068e1deb96767fbc9f7c493523" exitCode=60 Nov 02 21:30:04.228463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.228072 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" event={"ID":"df26296c-411c-4a5e-8261-d7e03612d9e3","Type":"ContainerDied","Data":"967e5369adf7ce13fa6233a1e83a3bca6da9d8068e1deb96767fbc9f7c493523"} Nov 02 21:30:04.228463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.228132 4441 scope.go:122] "RemoveContainer" containerID="430c90fb74e7864d9d4d416df201c027cce1cd580eeceab4d50cc215a3fc985d" Nov 02 21:30:04.228463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.228404 4441 scope.go:122] "RemoveContainer" containerID="967e5369adf7ce13fa6233a1e83a3bca6da9d8068e1deb96767fbc9f7c493523" Nov 02 21:30:04.228725 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:04.228705 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7_odh-ai-gateway-infra(df26296c-411c-4a5e-8261-d7e03612d9e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" Nov 02 21:30:04.229941 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.229914 4441 scope.go:122] "RemoveContainer" containerID="4d9604f804176a44388f3a844e8876abbb61d0e118b9092297319396bb6d31ab" Nov 02 21:30:04.230276 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:04.230229 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-7zh5g_odh-ai-gateway-infra(941bf58c-cda8-4103-9b66-9d45b0476bbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" Nov 02 21:30:04.231689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.231668 4441 generic.go:356] "Generic (PLEG): container finished" podID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerID="ba85618a5fd19a03858e4b9ccb48174a3c696724d4229f959868bb257e027e67" exitCode=60 Nov 02 21:30:04.231774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.231746 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" event={"ID":"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9","Type":"ContainerDied","Data":"ba85618a5fd19a03858e4b9ccb48174a3c696724d4229f959868bb257e027e67"} Nov 02 21:30:04.231915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.231902 4441 scope.go:122] "RemoveContainer" containerID="ba85618a5fd19a03858e4b9ccb48174a3c696724d4229f959868bb257e027e67" Nov 02 21:30:04.232197 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:04.232167 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj_odh-ai-gateway-infra(b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" Nov 02 21:30:04.236032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.236004 4441 generic.go:356] "Generic (PLEG): container finished" podID="27961744-e82b-4d10-9fae-5206f8a0d227" containerID="d2ef1a0b8a652ea53b8e897258bc5f011e46a8214c1b27e16aef5a4f4e7bd35d" exitCode=60 Nov 02 21:30:04.236200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.236178 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" event={"ID":"27961744-e82b-4d10-9fae-5206f8a0d227","Type":"ContainerDied","Data":"d2ef1a0b8a652ea53b8e897258bc5f011e46a8214c1b27e16aef5a4f4e7bd35d"} Nov 02 21:30:04.236289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.236276 4441 scope.go:122] "RemoveContainer" containerID="d2ef1a0b8a652ea53b8e897258bc5f011e46a8214c1b27e16aef5a4f4e7bd35d" Nov 02 21:30:04.236511 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:04.236483 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9_odh-ai-gateway-infra(27961744-e82b-4d10-9fae-5206f8a0d227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" Nov 02 21:30:04.238009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.237962 4441 generic.go:356] "Generic (PLEG): container finished" podID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerID="ded85024047be832880922c6258bbff089333c5a895641dca8f6ffdd229f8f1c" exitCode=60 Nov 02 21:30:04.238080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.238052 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" event={"ID":"4994be9a-363f-4f43-a0d4-b9e9bc8121d8","Type":"ContainerDied","Data":"ded85024047be832880922c6258bbff089333c5a895641dca8f6ffdd229f8f1c"} Nov 02 21:30:04.238252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.238235 4441 scope.go:122] "RemoveContainer" containerID="ded85024047be832880922c6258bbff089333c5a895641dca8f6ffdd229f8f1c" Nov 02 21:30:04.238515 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:04.238494 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt_odh-ai-gateway-infra(4994be9a-363f-4f43-a0d4-b9e9bc8121d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" Nov 02 21:30:04.250084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.250047 4441 scope.go:122] "RemoveContainer" containerID="4347eb590ff1a79e29ce29188f3102f802466371f7d8c74955359134bf144696" Nov 02 21:30:04.268407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.268386 4441 scope.go:122] "RemoveContainer" containerID="23f1dc6b2f39afc4394311b4c6eb6998439291733a590da2b1379950e2b74073" Nov 02 21:30:04.306610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.306591 4441 scope.go:122] "RemoveContainer" containerID="b9a7f7418b5663ff5b0f7581bf899198ee252705505983c32c0d6f6764d22226" Nov 02 21:30:04.902416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.902379 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:30:04.902581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:04.902432 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 21:30:05.247922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:05.247831 4441 scope.go:122] "RemoveContainer" containerID="967e5369adf7ce13fa6233a1e83a3bca6da9d8068e1deb96767fbc9f7c493523" Nov 02 21:30:05.248396 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:05.248158 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7_odh-ai-gateway-infra(df26296c-411c-4a5e-8261-d7e03612d9e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" Nov 02 21:30:05.249649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:05.249623 4441 scope.go:122] "RemoveContainer" containerID="ba85618a5fd19a03858e4b9ccb48174a3c696724d4229f959868bb257e027e67" Nov 02 21:30:05.249940 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:05.249918 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj_odh-ai-gateway-infra(b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" Nov 02 21:30:05.251242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:05.251223 4441 scope.go:122] "RemoveContainer" containerID="d2ef1a0b8a652ea53b8e897258bc5f011e46a8214c1b27e16aef5a4f4e7bd35d" Nov 02 21:30:05.251495 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:05.251473 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9_odh-ai-gateway-infra(27961744-e82b-4d10-9fae-5206f8a0d227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" Nov 02 21:30:05.252822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:05.252804 4441 scope.go:122] "RemoveContainer" containerID="ded85024047be832880922c6258bbff089333c5a895641dca8f6ffdd229f8f1c" Nov 02 21:30:05.253049 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:05.253031 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt_odh-ai-gateway-infra(4994be9a-363f-4f43-a0d4-b9e9bc8121d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" Nov 02 21:30:05.903002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:05.902945 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:30:05.903173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:05.903022 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 21:30:06.902335 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:06.902294 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:30:06.902785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:06.902364 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 21:30:07.903118 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:07.903086 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:30:07.903621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:07.903139 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 21:30:08.701073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:08.701032 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:08.902232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:08.902192 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:30:08.902408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:08.902260 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 21:30:09.906760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:09.906729 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:09.907213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:09.907192 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:09.907960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:09.907942 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 21:30:10.616836 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:10.616802 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:30:12.228749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.228703 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2"] Nov 02 21:30:12.232758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.232737 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.256598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.256574 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423c1021-69e2-4d17-849a-290277942c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.256720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.256699 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f947m\" (UniqueName: \"kubernetes.io/projected/423c1021-69e2-4d17-849a-290277942c76-kube-api-access-f947m\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.256773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.256742 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423c1021-69e2-4d17-849a-290277942c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.286390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.286368 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2"] Nov 02 21:30:12.327381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.327346 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:30:12.357448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.357421 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423c1021-69e2-4d17-849a-290277942c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.357569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.357553 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423c1021-69e2-4d17-849a-290277942c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.357695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.357680 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f947m\" (UniqueName: \"kubernetes.io/projected/423c1021-69e2-4d17-849a-290277942c76-kube-api-access-f947m\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.358140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.358119 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423c1021-69e2-4d17-849a-290277942c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.359896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.359877 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423c1021-69e2-4d17-849a-290277942c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.410943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.410914 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f947m\" (UniqueName: \"kubernetes.io/projected/423c1021-69e2-4d17-849a-290277942c76-kube-api-access-f947m\") pod \"maas-api-revoke-sub-90e09d91-jhpt2\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.546582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.546512 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:12.927238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:12.927205 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2"] Nov 02 21:30:12.931673 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:12.931644 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423c1021_69e2_4d17_849a_290277942c76.slice/crio-07a0c24535d6393a3787e2c2a16c854e763d340a73debd5a07e0e422dd6d69fe WatchSource:0}: Error finding container 07a0c24535d6393a3787e2c2a16c854e763d340a73debd5a07e0e422dd6d69fe: Status 404 returned error can't find the container with id 07a0c24535d6393a3787e2c2a16c854e763d340a73debd5a07e0e422dd6d69fe Nov 02 21:30:13.181993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:13.181944 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:13.356140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:13.356103 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" event={"ID":"423c1021-69e2-4d17-849a-290277942c76","Type":"ContainerStarted","Data":"5f2916631b8ff18f7154d081634004fff5bfe53361403cd1f0727f4cc99cd08a"} Nov 02 21:30:13.356527 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:13.356179 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" event={"ID":"423c1021-69e2-4d17-849a-290277942c76","Type":"ContainerStarted","Data":"07a0c24535d6393a3787e2c2a16c854e763d340a73debd5a07e0e422dd6d69fe"} Nov 02 21:30:13.464324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:13.464228 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" podStartSLOduration=1.464214202 podStartE2EDuration="1.464214202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:13.462424964 +0000 UTC m=+1043.018053301" watchObservedRunningTime="2026-11-02 21:30:13.464214202 +0000 UTC m=+1043.019842539" Nov 02 21:30:14.364758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:14.364719 4441 generic.go:356] "Generic (PLEG): container finished" podID="423c1021-69e2-4d17-849a-290277942c76" containerID="5f2916631b8ff18f7154d081634004fff5bfe53361403cd1f0727f4cc99cd08a" exitCode=0 Nov 02 21:30:14.365149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:14.364802 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" event={"ID":"423c1021-69e2-4d17-849a-290277942c76","Type":"ContainerDied","Data":"5f2916631b8ff18f7154d081634004fff5bfe53361403cd1f0727f4cc99cd08a"} Nov 02 21:30:14.577015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:14.576986 4441 scope.go:122] "RemoveContainer" containerID="4d9604f804176a44388f3a844e8876abbb61d0e118b9092297319396bb6d31ab" Nov 02 21:30:14.577184 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:14.577036 4441 scope.go:122] "RemoveContainer" containerID="ded85024047be832880922c6258bbff089333c5a895641dca8f6ffdd229f8f1c" Nov 02 21:30:15.377304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.377265 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" event={"ID":"4994be9a-363f-4f43-a0d4-b9e9bc8121d8","Type":"ContainerStarted","Data":"87e2456420a00a440504d0e28a747e0e00af7ac90798d4de656890e2a48b0533"} Nov 02 21:30:15.379711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.379682 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" event={"ID":"941bf58c-cda8-4103-9b66-9d45b0476bbe","Type":"ContainerStarted","Data":"21aaa1f37ec38c07c514e0036b93d5b3cae754c13e418f49f76cab6f0444fa4f"} Nov 02 21:30:15.576326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.576306 4441 scope.go:122] "RemoveContainer" containerID="967e5369adf7ce13fa6233a1e83a3bca6da9d8068e1deb96767fbc9f7c493523" Nov 02 21:30:15.592663 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.592615 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" podStartSLOduration=15.592602377 podStartE2EDuration="15.592602377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:15.590554594 +0000 UTC m=+1045.146182931" watchObservedRunningTime="2026-11-02 21:30:15.592602377 +0000 UTC m=+1045.148230713" Nov 02 21:30:15.634565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.634488 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:15.696455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.696413 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423c1021-69e2-4d17-849a-290277942c76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423c1021-69e2-4d17-849a-290277942c76-maas-api-xks-ca\") pod \"423c1021-69e2-4d17-849a-290277942c76\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " Nov 02 21:30:15.696647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.696500 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423c1021-69e2-4d17-849a-290277942c76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423c1021-69e2-4d17-849a-290277942c76-maas-api-openshift-ca\") pod \"423c1021-69e2-4d17-849a-290277942c76\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " Nov 02 21:30:15.696710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.696663 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423c1021-69e2-4d17-849a-290277942c76-kube-api-access-f947m\" (UniqueName: \"kubernetes.io/projected/423c1021-69e2-4d17-849a-290277942c76-kube-api-access-f947m\") pod \"423c1021-69e2-4d17-849a-290277942c76\" (UID: \"423c1021-69e2-4d17-849a-290277942c76\") " Nov 02 21:30:15.696854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.696821 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423c1021-69e2-4d17-849a-290277942c76-maas-api-openshift-ca" pod "423c1021-69e2-4d17-849a-290277942c76" (UID: "423c1021-69e2-4d17-849a-290277942c76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:15.697144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.697109 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423c1021-69e2-4d17-849a-290277942c76-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.698684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.698656 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423c1021-69e2-4d17-849a-290277942c76-maas-api-xks-ca" pod "423c1021-69e2-4d17-849a-290277942c76" (UID: "423c1021-69e2-4d17-849a-290277942c76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:30:15.698775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.698752 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423c1021-69e2-4d17-849a-290277942c76-kube-api-access-f947m" pod "423c1021-69e2-4d17-849a-290277942c76" (UID: "423c1021-69e2-4d17-849a-290277942c76"). InnerVolumeSpecName "kube-api-access-f947m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:15.815243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.801222 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423c1021-69e2-4d17-849a-290277942c76-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.815243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:15.801257 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f947m\" (UniqueName: \"kubernetes.io/projected/423c1021-69e2-4d17-849a-290277942c76-kube-api-access-f947m\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:16.397588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.394958 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" event={"ID":"df26296c-411c-4a5e-8261-d7e03612d9e3","Type":"ContainerStarted","Data":"29e5a29b535fd30bd5d72cc429832efdc8eb7f0ce20215ca250f8b606b25f6dc"} Nov 02 21:30:16.404607 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.404576 4441 generic.go:356] "Generic (PLEG): container finished" podID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerID="21aaa1f37ec38c07c514e0036b93d5b3cae754c13e418f49f76cab6f0444fa4f" exitCode=60 Nov 02 21:30:16.405003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.404981 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" event={"ID":"941bf58c-cda8-4103-9b66-9d45b0476bbe","Type":"ContainerDied","Data":"21aaa1f37ec38c07c514e0036b93d5b3cae754c13e418f49f76cab6f0444fa4f"} Nov 02 21:30:16.405163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.405151 4441 scope.go:122] "RemoveContainer" containerID="4d9604f804176a44388f3a844e8876abbb61d0e118b9092297319396bb6d31ab" Nov 02 21:30:16.405691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.405672 4441 scope.go:122] "RemoveContainer" containerID="21aaa1f37ec38c07c514e0036b93d5b3cae754c13e418f49f76cab6f0444fa4f" Nov 02 21:30:16.406139 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:16.406114 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-7zh5g_odh-ai-gateway-infra(941bf58c-cda8-4103-9b66-9d45b0476bbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" Nov 02 21:30:16.409554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.409514 4441 generic.go:356] "Generic (PLEG): container finished" podID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerID="87e2456420a00a440504d0e28a747e0e00af7ac90798d4de656890e2a48b0533" exitCode=60 Nov 02 21:30:16.409708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.409688 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" event={"ID":"4994be9a-363f-4f43-a0d4-b9e9bc8121d8","Type":"ContainerDied","Data":"87e2456420a00a440504d0e28a747e0e00af7ac90798d4de656890e2a48b0533"} Nov 02 21:30:16.410281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.410263 4441 scope.go:122] "RemoveContainer" containerID="87e2456420a00a440504d0e28a747e0e00af7ac90798d4de656890e2a48b0533" Nov 02 21:30:16.410674 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:16.410652 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt_odh-ai-gateway-infra(4994be9a-363f-4f43-a0d4-b9e9bc8121d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" Nov 02 21:30:16.413502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.412479 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" event={"ID":"423c1021-69e2-4d17-849a-290277942c76","Type":"ContainerDied","Data":"07a0c24535d6393a3787e2c2a16c854e763d340a73debd5a07e0e422dd6d69fe"} Nov 02 21:30:16.413502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.412531 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a0c24535d6393a3787e2c2a16c854e763d340a73debd5a07e0e422dd6d69fe" Nov 02 21:30:16.413502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.412646 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2" Nov 02 21:30:16.451005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.450349 4441 scope.go:122] "RemoveContainer" containerID="ded85024047be832880922c6258bbff089333c5a895641dca8f6ffdd229f8f1c" Nov 02 21:30:16.510538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.509946 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt"] Nov 02 21:30:16.601713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.586560 4441 scope.go:122] "RemoveContainer" containerID="ba85618a5fd19a03858e4b9ccb48174a3c696724d4229f959868bb257e027e67" Nov 02 21:30:16.601713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.589092 4441 scope.go:122] "RemoveContainer" containerID="d2ef1a0b8a652ea53b8e897258bc5f011e46a8214c1b27e16aef5a4f4e7bd35d" Nov 02 21:30:16.677519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:16.647442 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g"] Nov 02 21:30:17.421861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.421831 4441 generic.go:356] "Generic (PLEG): container finished" podID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerID="29e5a29b535fd30bd5d72cc429832efdc8eb7f0ce20215ca250f8b606b25f6dc" exitCode=60 Nov 02 21:30:17.422343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.421899 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" event={"ID":"df26296c-411c-4a5e-8261-d7e03612d9e3","Type":"ContainerDied","Data":"29e5a29b535fd30bd5d72cc429832efdc8eb7f0ce20215ca250f8b606b25f6dc"} Nov 02 21:30:17.422343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.421962 4441 scope.go:122] "RemoveContainer" containerID="967e5369adf7ce13fa6233a1e83a3bca6da9d8068e1deb96767fbc9f7c493523" Nov 02 21:30:17.422465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.422377 4441 scope.go:122] "RemoveContainer" containerID="29e5a29b535fd30bd5d72cc429832efdc8eb7f0ce20215ca250f8b606b25f6dc" Nov 02 21:30:17.422683 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:17.422662 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7_odh-ai-gateway-infra(df26296c-411c-4a5e-8261-d7e03612d9e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" Nov 02 21:30:17.426563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.426541 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" event={"ID":"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9","Type":"ContainerStarted","Data":"7e36dd61d70a524bb646d30f0c09d092979d14bcff6fb38a038166b885c02995"} Nov 02 21:30:17.428736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.428676 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" event={"ID":"27961744-e82b-4d10-9fae-5206f8a0d227","Type":"ContainerStarted","Data":"f793f9fd57b737f5f2bbbb0210297984071f39ff96462e46835c164bbae7b34d"} Nov 02 21:30:17.519760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.519728 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7"] Nov 02 21:30:17.603363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.603338 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:17.616408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.616385 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:17.663347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.663322 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-maas-api-service-ca\") pod \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " Nov 02 21:30:17.663467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.663365 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941bf58c-cda8-4103-9b66-9d45b0476bbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941bf58c-cda8-4103-9b66-9d45b0476bbe-maas-api-service-ca\") pod \"941bf58c-cda8-4103-9b66-9d45b0476bbe\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " Nov 02 21:30:17.663467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.663439 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941bf58c-cda8-4103-9b66-9d45b0476bbe-kube-api-access-kwb27\" (UniqueName: \"kubernetes.io/projected/941bf58c-cda8-4103-9b66-9d45b0476bbe-kube-api-access-kwb27\") pod \"941bf58c-cda8-4103-9b66-9d45b0476bbe\" (UID: \"941bf58c-cda8-4103-9b66-9d45b0476bbe\") " Nov 02 21:30:17.663467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.663458 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-kube-api-access-4gq9d\" (UniqueName: \"kubernetes.io/projected/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-kube-api-access-4gq9d\") pod \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\" (UID: \"4994be9a-363f-4f43-a0d4-b9e9bc8121d8\") " Nov 02 21:30:17.663727 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.663699 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-maas-api-service-ca" pod "4994be9a-363f-4f43-a0d4-b9e9bc8121d8" (UID: "4994be9a-363f-4f43-a0d4-b9e9bc8121d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:17.663871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.663849 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941bf58c-cda8-4103-9b66-9d45b0476bbe-maas-api-service-ca" pod "941bf58c-cda8-4103-9b66-9d45b0476bbe" (UID: "941bf58c-cda8-4103-9b66-9d45b0476bbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:17.665563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.665538 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941bf58c-cda8-4103-9b66-9d45b0476bbe-kube-api-access-kwb27" pod "941bf58c-cda8-4103-9b66-9d45b0476bbe" (UID: "941bf58c-cda8-4103-9b66-9d45b0476bbe"). InnerVolumeSpecName "kube-api-access-kwb27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:17.665655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.665572 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-kube-api-access-4gq9d" pod "4994be9a-363f-4f43-a0d4-b9e9bc8121d8" (UID: "4994be9a-363f-4f43-a0d4-b9e9bc8121d8"). InnerVolumeSpecName "kube-api-access-4gq9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:17.704438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.704380 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:17.767642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.767605 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwb27\" (UniqueName: \"kubernetes.io/projected/941bf58c-cda8-4103-9b66-9d45b0476bbe-kube-api-access-kwb27\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:17.767642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.767641 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gq9d\" (UniqueName: \"kubernetes.io/projected/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-kube-api-access-4gq9d\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:17.768047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.767656 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994be9a-363f-4f43-a0d4-b9e9bc8121d8-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:17.768047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:17.767672 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941bf58c-cda8-4103-9b66-9d45b0476bbe-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:18.440007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.439950 4441 generic.go:356] "Generic (PLEG): container finished" podID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerID="7e36dd61d70a524bb646d30f0c09d092979d14bcff6fb38a038166b885c02995" exitCode=60 Nov 02 21:30:18.440470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.440028 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" event={"ID":"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9","Type":"ContainerDied","Data":"7e36dd61d70a524bb646d30f0c09d092979d14bcff6fb38a038166b885c02995"} Nov 02 21:30:18.440470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.440088 4441 scope.go:122] "RemoveContainer" containerID="ba85618a5fd19a03858e4b9ccb48174a3c696724d4229f959868bb257e027e67" Nov 02 21:30:18.440616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.440482 4441 scope.go:122] "RemoveContainer" containerID="7e36dd61d70a524bb646d30f0c09d092979d14bcff6fb38a038166b885c02995" Nov 02 21:30:18.440839 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:18.440816 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj_odh-ai-gateway-infra(b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" Nov 02 21:30:18.442289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.442266 4441 generic.go:356] "Generic (PLEG): container finished" podID="27961744-e82b-4d10-9fae-5206f8a0d227" containerID="f793f9fd57b737f5f2bbbb0210297984071f39ff96462e46835c164bbae7b34d" exitCode=60 Nov 02 21:30:18.442374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.442335 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" event={"ID":"27961744-e82b-4d10-9fae-5206f8a0d227","Type":"ContainerDied","Data":"f793f9fd57b737f5f2bbbb0210297984071f39ff96462e46835c164bbae7b34d"} Nov 02 21:30:18.442666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.442650 4441 scope.go:122] "RemoveContainer" containerID="f793f9fd57b737f5f2bbbb0210297984071f39ff96462e46835c164bbae7b34d" Nov 02 21:30:18.442958 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:18.442933 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9_odh-ai-gateway-infra(27961744-e82b-4d10-9fae-5206f8a0d227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" Nov 02 21:30:18.443708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.443685 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" Nov 02 21:30:18.443774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.443723 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt" event={"ID":"4994be9a-363f-4f43-a0d4-b9e9bc8121d8","Type":"ContainerDied","Data":"15e529f05be6b0a413c2c1ef9da00d955a671a3cbb1aeb8108b976091cfa3032"} Nov 02 21:30:18.447245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.447225 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" Nov 02 21:30:18.447348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.447249 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g" event={"ID":"941bf58c-cda8-4103-9b66-9d45b0476bbe","Type":"ContainerDied","Data":"edae59eb72d84b49eb41f2ff1fc06d5493a98a00c204060a6b745bddc0212a37"} Nov 02 21:30:18.456438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.456422 4441 scope.go:122] "RemoveContainer" containerID="d2ef1a0b8a652ea53b8e897258bc5f011e46a8214c1b27e16aef5a4f4e7bd35d" Nov 02 21:30:18.469849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.469832 4441 scope.go:122] "RemoveContainer" containerID="87e2456420a00a440504d0e28a747e0e00af7ac90798d4de656890e2a48b0533" Nov 02 21:30:18.483117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.482219 4441 scope.go:122] "RemoveContainer" containerID="21aaa1f37ec38c07c514e0036b93d5b3cae754c13e418f49f76cab6f0444fa4f" Nov 02 21:30:18.594877 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.594849 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:18.655885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.655856 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt"] Nov 02 21:30:18.660474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.660452 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894250-tv7wt"] Nov 02 21:30:18.677304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.677277 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df26296c-411c-4a5e-8261-d7e03612d9e3-kube-api-access-wbf8d\" (UniqueName: \"kubernetes.io/projected/df26296c-411c-4a5e-8261-d7e03612d9e3-kube-api-access-wbf8d\") pod \"df26296c-411c-4a5e-8261-d7e03612d9e3\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " Nov 02 21:30:18.677432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.677346 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df26296c-411c-4a5e-8261-d7e03612d9e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df26296c-411c-4a5e-8261-d7e03612d9e3-maas-api-service-ca\") pod \"df26296c-411c-4a5e-8261-d7e03612d9e3\" (UID: \"df26296c-411c-4a5e-8261-d7e03612d9e3\") " Nov 02 21:30:18.677685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.677661 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df26296c-411c-4a5e-8261-d7e03612d9e3-maas-api-service-ca" pod "df26296c-411c-4a5e-8261-d7e03612d9e3" (UID: "df26296c-411c-4a5e-8261-d7e03612d9e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:18.679433 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.679414 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df26296c-411c-4a5e-8261-d7e03612d9e3-kube-api-access-wbf8d" pod "df26296c-411c-4a5e-8261-d7e03612d9e3" (UID: "df26296c-411c-4a5e-8261-d7e03612d9e3"). InnerVolumeSpecName "kube-api-access-wbf8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:18.769175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.769087 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj"] Nov 02 21:30:18.779030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.779000 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df26296c-411c-4a5e-8261-d7e03612d9e3-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:18.779030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.779029 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbf8d\" (UniqueName: \"kubernetes.io/projected/df26296c-411c-4a5e-8261-d7e03612d9e3-kube-api-access-wbf8d\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:18.974612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.974578 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g"] Nov 02 21:30:18.979014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.978985 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-7zh5g"] Nov 02 21:30:18.984615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:18.984587 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9"] Nov 02 21:30:19.459752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.459731 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" Nov 02 21:30:19.459752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.459736 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7" event={"ID":"df26296c-411c-4a5e-8261-d7e03612d9e3","Type":"ContainerDied","Data":"aef6c5988cf3487826ed716bfe2f2664d681ee55179fca3bf3e1e746b39a02b2"} Nov 02 21:30:19.460164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.459799 4441 scope.go:122] "RemoveContainer" containerID="29e5a29b535fd30bd5d72cc429832efdc8eb7f0ce20215ca250f8b606b25f6dc" Nov 02 21:30:19.645632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.645608 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:19.648264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.648245 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:19.688636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.688608 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-kube-api-access-2hwf8\" (UniqueName: \"kubernetes.io/projected/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-kube-api-access-2hwf8\") pod \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " Nov 02 21:30:19.688747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.688716 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27961744-e82b-4d10-9fae-5206f8a0d227-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27961744-e82b-4d10-9fae-5206f8a0d227-maas-api-service-ca\") pod \"27961744-e82b-4d10-9fae-5206f8a0d227\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " Nov 02 21:30:19.688830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.688813 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-maas-api-service-ca\") pod \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\" (UID: \"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9\") " Nov 02 21:30:19.688902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.688887 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27961744-e82b-4d10-9fae-5206f8a0d227-kube-api-access-klv9r\" (UniqueName: \"kubernetes.io/projected/27961744-e82b-4d10-9fae-5206f8a0d227-kube-api-access-klv9r\") pod \"27961744-e82b-4d10-9fae-5206f8a0d227\" (UID: \"27961744-e82b-4d10-9fae-5206f8a0d227\") " Nov 02 21:30:19.689173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.689145 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27961744-e82b-4d10-9fae-5206f8a0d227-maas-api-service-ca" pod "27961744-e82b-4d10-9fae-5206f8a0d227" (UID: "27961744-e82b-4d10-9fae-5206f8a0d227"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:19.689290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.689192 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-maas-api-service-ca" pod "b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" (UID: "b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:19.689356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.689329 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27961744-e82b-4d10-9fae-5206f8a0d227-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:19.689356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.689350 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:19.691377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.691350 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-kube-api-access-2hwf8" pod "b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" (UID: "b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9"). InnerVolumeSpecName "kube-api-access-2hwf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:19.691461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.691382 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27961744-e82b-4d10-9fae-5206f8a0d227-kube-api-access-klv9r" pod "27961744-e82b-4d10-9fae-5206f8a0d227" (UID: "27961744-e82b-4d10-9fae-5206f8a0d227"). InnerVolumeSpecName "kube-api-access-klv9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:19.779543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.779469 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7"] Nov 02 21:30:19.786313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.786287 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894250-ms2g7"] Nov 02 21:30:19.790533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.790510 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klv9r\" (UniqueName: \"kubernetes.io/projected/27961744-e82b-4d10-9fae-5206f8a0d227-kube-api-access-klv9r\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:19.790586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:19.790540 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwf8\" (UniqueName: \"kubernetes.io/projected/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9-kube-api-access-2hwf8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:20.471205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.471168 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" Nov 02 21:30:20.471205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.471181 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj" event={"ID":"b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9","Type":"ContainerDied","Data":"de0e28f69da1ed636b13532daf49417851c82ace5a668c04396e59428aeccacf"} Nov 02 21:30:20.471762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.471250 4441 scope.go:122] "RemoveContainer" containerID="7e36dd61d70a524bb646d30f0c09d092979d14bcff6fb38a038166b885c02995" Nov 02 21:30:20.473095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.473065 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" Nov 02 21:30:20.473230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.473110 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9" event={"ID":"27961744-e82b-4d10-9fae-5206f8a0d227","Type":"ContainerDied","Data":"66058beef3146fc27d37244fb4b02716146be8ce169bfb525c2c05f094bab5e4"} Nov 02 21:30:20.483215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.483193 4441 scope.go:122] "RemoveContainer" containerID="f793f9fd57b737f5f2bbbb0210297984071f39ff96462e46835c164bbae7b34d" Nov 02 21:30:20.585414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.585386 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" path="/var/lib/kubelet/pods/4994be9a-363f-4f43-a0d4-b9e9bc8121d8/volumes" Nov 02 21:30:20.585800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.585787 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" path="/var/lib/kubelet/pods/941bf58c-cda8-4103-9b66-9d45b0476bbe/volumes" Nov 02 21:30:20.586189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.586175 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" path="/var/lib/kubelet/pods/df26296c-411c-4a5e-8261-d7e03612d9e3/volumes" Nov 02 21:30:20.628326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.628296 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj"] Nov 02 21:30:20.675207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.675180 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dd6574-29894250-trpcj"] Nov 02 21:30:20.778146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.778058 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9"] Nov 02 21:30:20.788011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:20.787982 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4d9ea0-29894250-wcjv9"] Nov 02 21:30:22.288185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:22.288155 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:22.583936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:22.583906 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" path="/var/lib/kubelet/pods/27961744-e82b-4d10-9fae-5206f8a0d227/volumes" Nov 02 21:30:22.584359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:22.584344 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" path="/var/lib/kubelet/pods/b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9/volumes" Nov 02 21:30:26.786161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:26.786126 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:31.331328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.331282 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:31.513442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.513407 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh"] Nov 02 21:30:31.514109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514083 4441 cpu_manager.go:414] "containerMap: removing container" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:30:31.514109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514106 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514121 4441 cpu_manager.go:414] "containerMap: removing container" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514130 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514144 4441 cpu_manager.go:414] "containerMap: removing container" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514152 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514167 4441 cpu_manager.go:414] "containerMap: removing container" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514175 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514185 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514192 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514202 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514207 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514216 4441 cpu_manager.go:414] "containerMap: removing container" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514220 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514231 4441 cpu_manager.go:414] "containerMap: removing container" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514238 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514255 4441 cpu_manager.go:414] "containerMap: removing container" podUID="423c1021-69e2-4d17-849a-290277942c76" containerName="revoke-keys" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514260 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423c1021-69e2-4d17-849a-290277942c76" containerName="revoke-keys" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514283 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514288 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514294 4441 cpu_manager.go:414] "containerMap: removing container" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514298 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514303 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.514305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514308 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514318 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514325 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514429 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514438 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514444 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514450 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514456 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="423c1021-69e2-4d17-849a-290277942c76" containerName="revoke-keys" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514464 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514469 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514473 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514478 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514484 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514488 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514494 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:30:31.515089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.514499 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:30:31.518214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.518194 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.573651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.573623 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh"] Nov 02 21:30:31.615140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615070 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4e2b27-openshift-default-dockercfg-knh48\"" Nov 02 21:30:31.615589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615561 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.615647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615633 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.615691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615655 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzpp\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-kube-api-access-jlzpp\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.615735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615700 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.615781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615735 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.615836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615818 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.615899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.615884 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.616043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.616017 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.616131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.616075 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717500 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717546 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717571 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzpp\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-kube-api-access-jlzpp\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717598 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717639 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717678 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717704 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717768 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.717959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.717817 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.718235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.718168 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.718315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.718266 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.718315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.718289 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.718449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.718424 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.718513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.718494 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.719875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.719850 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.720201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.720182 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.771627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.771601 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.771795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.771778 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzpp\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-kube-api-access-jlzpp\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:31.832722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:31.832699 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:32.269947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.269868 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh"] Nov 02 21:30:32.274534 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:30:32.274500 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb19e54e_4bcb_4f51_b8c6_a1414e998ede.slice/crio-55aed281397223432e283d6dd8b68f41fe37ab67b7c125967d174d9e5d784058 WatchSource:0}: Error finding container 55aed281397223432e283d6dd8b68f41fe37ab67b7c125967d174d9e5d784058: Status 404 returned error can't find the container with id 55aed281397223432e283d6dd8b68f41fe37ab67b7c125967d174d9e5d784058 Nov 02 21:30:32.276980 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.276936 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:30:32.277061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.277027 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:30:32.277096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.277061 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:30:32.568453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.568413 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" event={"ID":"cb19e54e-4bcb-4f51-b8c6-a1414e998ede","Type":"ContainerStarted","Data":"050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a"} Nov 02 21:30:32.568934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.568475 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" event={"ID":"cb19e54e-4bcb-4f51-b8c6-a1414e998ede","Type":"ContainerStarted","Data":"55aed281397223432e283d6dd8b68f41fe37ab67b7c125967d174d9e5d784058"} Nov 02 21:30:32.684669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.684609 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podStartSLOduration=1.684594627 podStartE2EDuration="1.684594627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:32.682001742 +0000 UTC m=+1062.237630080" watchObservedRunningTime="2026-11-02 21:30:32.684594627 +0000 UTC m=+1062.240222963" Nov 02 21:30:32.832923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:32.832823 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:33.834397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:33.834362 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:30:33.834794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:33.834442 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 21:30:34.833337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:34.833298 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:30:34.833509 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:34.833351 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 21:30:35.833794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:35.833757 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:30:35.834215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:35.833813 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 21:30:35.913285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:35.913255 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:36.833519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:36.833483 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:30:36.833707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:36.833548 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 21:30:37.833197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:37.833160 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:30:37.833682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:37.833221 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 21:30:38.833226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:38.833190 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:30:38.833634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:38.833245 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 21:30:39.837705 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:39.837679 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:39.838174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:39.838155 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:39.838780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:39.838764 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:30:40.398585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:40.398549 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:40.616323 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:30:40.616281 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:30:42.461448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:42.461417 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:44.887026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:44.886993 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:46.372024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:46.371987 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:46.490251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:46.490215 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:49.399654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:49.399615 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:52.259473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:52.259436 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:54.029262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:54.029219 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:58.651831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:30:58.651801 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:01.657928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:01.657886 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:02.324393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.324355 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6"] Nov 02 21:31:02.324925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.324911 4441 cpu_manager.go:414] "containerMap: removing container" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:31:02.324997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.324927 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941bf58c-cda8-4103-9b66-9d45b0476bbe" containerName="cleanup" Nov 02 21:31:02.324997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.324940 4441 cpu_manager.go:414] "containerMap: removing container" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:31:02.324997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.324944 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:31:02.324997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.324959 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:31:02.324997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.324964 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4994be9a-363f-4f43-a0d4-b9e9bc8121d8" containerName="cleanup" Nov 02 21:31:02.325154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.325083 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bc0aeb-c8ca-4ed4-afe4-217c6085c1b9" containerName="cleanup" Nov 02 21:31:02.325154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.325091 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="df26296c-411c-4a5e-8261-d7e03612d9e3" containerName="cleanup" Nov 02 21:31:02.325154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.325100 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="27961744-e82b-4d10-9fae-5206f8a0d227" containerName="cleanup" Nov 02 21:31:02.330144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.330124 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374246 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374303 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374382 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374423 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22gj\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-kube-api-access-p22gj\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374450 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374466 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bcec41-697d-40da-be0c-0c31d967d59f-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374604 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374675 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.374808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.374791 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bcec41-697d-40da-be0c-0c31d967d59f-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.385859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.385831 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6"] Nov 02 21:31:02.475478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475444 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22gj\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-kube-api-access-p22gj\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475483 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475500 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bcec41-697d-40da-be0c-0c31d967d59f-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475554 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475592 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475687 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bcec41-697d-40da-be0c-0c31d967d59f-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475762 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475826 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.475923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475858 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.476113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.475943 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.476113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.476033 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.476225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.476203 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.476421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.476395 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.476664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.476637 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bcec41-697d-40da-be0c-0c31d967d59f-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.478363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.478331 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.478753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.478733 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bcec41-697d-40da-be0c-0c31d967d59f-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.529266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.529237 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.529795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.529772 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22gj\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-kube-api-access-p22gj\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:02.643458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:02.643361 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:03.087996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:03.087950 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6"] Nov 02 21:31:03.092305 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:31:03.092279 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bcec41_697d_40da_be0c_0c31d967d59f.slice/crio-f01ce8ef9fe33d9db2202c020e85b0c294edd1b4ba37a863f9dcb8df71e55a97 WatchSource:0}: Error finding container f01ce8ef9fe33d9db2202c020e85b0c294edd1b4ba37a863f9dcb8df71e55a97: Status 404 returned error can't find the container with id f01ce8ef9fe33d9db2202c020e85b0c294edd1b4ba37a863f9dcb8df71e55a97 Nov 02 21:31:03.094657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:03.094625 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:31:03.094754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:03.094693 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:31:03.094754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:03.094720 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:31:03.112359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:03.112336 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:03.835563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:03.835528 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" event={"ID":"37bcec41-697d-40da-be0c-0c31d967d59f","Type":"ContainerStarted","Data":"ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8"} Nov 02 21:31:03.835773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:03.835578 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" event={"ID":"37bcec41-697d-40da-be0c-0c31d967d59f","Type":"ContainerStarted","Data":"f01ce8ef9fe33d9db2202c020e85b0c294edd1b4ba37a863f9dcb8df71e55a97"} Nov 02 21:31:04.051044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:04.050990 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podStartSLOduration=2.050977002 podStartE2EDuration="2.050977002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:04.002563809 +0000 UTC m=+1093.558192146" watchObservedRunningTime="2026-11-02 21:31:04.050977002 +0000 UTC m=+1093.606605330" Nov 02 21:31:04.643859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:04.643823 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:04.645733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:04.645706 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:31:04.645864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:04.645753 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 21:31:05.644227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:05.644185 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:31:05.644732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:05.644247 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 21:31:06.643728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:06.643687 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:31:06.643904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:06.643743 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 21:31:07.644202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:07.644169 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:31:07.644698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:07.644228 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 21:31:07.693990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:07.693601 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:08.536566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:08.536506 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:08.644902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:08.644862 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:31:08.645330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:08.644934 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 21:31:09.644423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:09.644388 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:31:09.644671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:09.644453 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 21:31:10.381570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:10.381524 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:10.617134 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:31:10.617097 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:31:10.648401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:10.648329 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:10.648812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:10.648793 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:10.649411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:10.649394 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:31:12.302343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:12.302299 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:16.778741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:16.778705 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:19.789047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:19.789005 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:19.938876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:19.938830 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:21.309802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:21.309766 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:25.787068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:25.787037 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:30.284503 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:30.284458 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:34.776384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:34.776342 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:39.294888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:39.294847 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:40.616752 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:31:40.616716 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:31:43.331136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:43.331102 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:43.905139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:43.905102 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:46.777682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:46.777641 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:48.444093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:48.444053 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:53.246682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:53.246645 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:57.153773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.153744 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks"] Nov 02 21:31:57.157751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.157731 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.213538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.213502 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks"] Nov 02 21:31:57.229687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.229652 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:57.231874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.231844 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-home\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.231874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.231875 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-dshm\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.232108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.231910 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-model-cache\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.232108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.232089 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tmp-dir\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.232227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.232157 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tls-certs\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.232227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.232207 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmnw\" (UniqueName: \"kubernetes.io/projected/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-kube-api-access-gqmnw\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.253644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.253624 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dd6574\"/\"distinct-2-w0-dd6574-kserve-self-signed-certs\"" Nov 02 21:31:57.333514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.333480 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tmp-dir\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.333684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.333572 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tls-certs\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.333684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.333597 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmnw\" (UniqueName: \"kubernetes.io/projected/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-kube-api-access-gqmnw\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.333684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.333665 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-home\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.333847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.333686 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-dshm\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.333903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.333834 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-model-cache\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.334009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.333955 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tmp-dir\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.334108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.334089 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-home\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.334239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.334220 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-model-cache\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.335827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.335811 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-dshm\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.336145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.336127 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tls-certs\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.387410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.387376 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmnw\" (UniqueName: \"kubernetes.io/projected/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-kube-api-access-gqmnw\") pod \"distinct-2-w0-dd6574-kserve-7896b88cf6-79xks\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.470460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.470369 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:31:57.837557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.837497 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:57.885485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:57.885418 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks"] Nov 02 21:31:58.304057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:58.303946 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" event={"ID":"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f","Type":"ContainerStarted","Data":"55b6eaf9a80c880057051c92b1b27e535bdd4229c7ba2383a16a931562fd3aee"} Nov 02 21:31:58.304057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:58.304014 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" event={"ID":"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f","Type":"ContainerStarted","Data":"85b169daf970ac8e4728d270cb0bcfcd62ad29d4f031917d21dc33363f743541"} Nov 02 21:31:58.417345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:58.417292 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podStartSLOduration=1.4172749869999999 podStartE2EDuration="1.417274987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:58.412302689 +0000 UTC m=+1147.967931031" watchObservedRunningTime="2026-11-02 21:31:58.417274987 +0000 UTC m=+1147.972903325" Nov 02 21:31:58.470755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:31:58.470711 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:32:02.201174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.201134 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d"] Nov 02 21:32:02.205268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.205245 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.262015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.261961 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d"] Nov 02 21:32:02.301281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301244 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:02.301281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301261 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fbb463-48a0-41fa-877e-1bbaab353269-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301339 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301452 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrvv\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-kube-api-access-ljrvv\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301519 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fbb463-48a0-41fa-877e-1bbaab353269-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301602 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301656 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301631 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301658 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301759 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.301855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.301835 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.403660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.403624 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fbb463-48a0-41fa-877e-1bbaab353269-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.403857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.403678 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.403857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.403704 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.403857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.403728 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.403857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.403774 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.403857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.403824 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.404167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.403904 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fbb463-48a0-41fa-877e-1bbaab353269-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.404167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.404017 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.404167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.404070 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrvv\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-kube-api-access-ljrvv\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.404376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.404349 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.404534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.404507 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fbb463-48a0-41fa-877e-1bbaab353269-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.404626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.404599 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.404908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.404889 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.405039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.404935 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.406635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.406608 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.406750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.406727 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fbb463-48a0-41fa-877e-1bbaab353269-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.457626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.457563 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.457804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.457779 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrvv\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-kube-api-access-ljrvv\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.520588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.520550 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:02.954647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.954609 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d"] Nov 02 21:32:02.959139 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:32:02.959103 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fbb463_48a0_41fa_877e_1bbaab353269.slice/crio-3363c857ae21a2bdbd3e4d27fd044f50cb081b346b57619ba747cce8f91c1d46 WatchSource:0}: Error finding container 3363c857ae21a2bdbd3e4d27fd044f50cb081b346b57619ba747cce8f91c1d46: Status 404 returned error can't find the container with id 3363c857ae21a2bdbd3e4d27fd044f50cb081b346b57619ba747cce8f91c1d46 Nov 02 21:32:02.962352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.962311 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:32:02.962444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.962400 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:32:02.962511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:02.962444 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:32:03.351491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:03.351441 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" event={"ID":"59fbb463-48a0-41fa-877e-1bbaab353269","Type":"ContainerStarted","Data":"398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75"} Nov 02 21:32:03.352063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:03.351513 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" event={"ID":"59fbb463-48a0-41fa-877e-1bbaab353269","Type":"ContainerStarted","Data":"3363c857ae21a2bdbd3e4d27fd044f50cb081b346b57619ba747cce8f91c1d46"} Nov 02 21:32:03.521644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:03.521604 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:03.558215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:03.558155 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podStartSLOduration=1.558137772 podStartE2EDuration="1.558137772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:03.50853539 +0000 UTC m=+1153.064163728" watchObservedRunningTime="2026-11-02 21:32:03.558137772 +0000 UTC m=+1153.113766110" Nov 02 21:32:04.523445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:04.523413 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:32:04.523898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:04.523473 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 21:32:05.521306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:05.521271 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:32:05.521516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:05.521323 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 21:32:06.520853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:06.520818 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:32:06.521261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:06.520871 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 21:32:07.201601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:07.201559 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:07.484409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:07.484321 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:32:07.505498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:07.505463 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:32:07.521490 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:07.521460 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:32:07.521806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:07.521514 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 21:32:08.432944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:08.432916 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:32:08.521598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:08.521563 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:32:08.522018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:08.521624 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 21:32:09.521623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:09.521588 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:32:09.522110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:09.521646 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 21:32:10.526332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:10.526302 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:10.526836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:10.526811 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:10.527760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:10.527738 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:32:10.615166 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:32:10.615135 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:32:11.893785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:11.893753 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:16.488032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:16.487998 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:21.100019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:21.099962 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:25.591610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:25.591577 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:30.102545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.102501 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:30.821287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.821255 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6"] Nov 02 21:32:30.825813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.825793 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:30.881144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.881117 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6"] Nov 02 21:32:30.911430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.911399 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:30.920494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.920459 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"distinct-2-w6-284b8e-kserve-self-signed-certs\"" Nov 02 21:32:30.969959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.969907 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-dshm\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:30.970364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.970335 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdsfp\" (UniqueName: \"kubernetes.io/projected/36377f3c-6f41-4ce3-8778-59f87630e3ab-kube-api-access-hdsfp\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:30.970516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.970499 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-model-cache\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:30.970740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.970724 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36377f3c-6f41-4ce3-8778-59f87630e3ab-tls-certs\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:30.992398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.992346 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-home\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:30.992398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:30.992402 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-tmp-dir\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094094 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdsfp\" (UniqueName: \"kubernetes.io/projected/36377f3c-6f41-4ce3-8778-59f87630e3ab-kube-api-access-hdsfp\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094149 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-model-cache\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094269 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36377f3c-6f41-4ce3-8778-59f87630e3ab-tls-certs\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094323 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-home\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094351 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-tmp-dir\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094423 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-dshm\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094640 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-model-cache\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094776 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-home\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.094862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.094796 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-tmp-dir\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.096763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.096732 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-dshm\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.097091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.097071 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36377f3c-6f41-4ce3-8778-59f87630e3ab-tls-certs\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.148064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.148031 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdsfp\" (UniqueName: \"kubernetes.io/projected/36377f3c-6f41-4ce3-8778-59f87630e3ab-kube-api-access-hdsfp\") pod \"distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.438234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.438153 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:31.532674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.532646 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj"] Nov 02 21:32:31.538604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.538575 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.702665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.702527 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.702665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.702584 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.703065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.703042 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.703160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.703102 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.703239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.703212 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.703332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.703317 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.703411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.703387 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.703453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.703437 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwznl\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-kube-api-access-rwznl\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.703600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.703585 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.730559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.730531 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj"] Nov 02 21:32:31.804262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804231 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.804417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804329 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.804417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804361 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.804560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804419 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.804560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804488 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.804560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804554 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.804709 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804598 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.804849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804813 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.805003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804873 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwznl\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-kube-api-access-rwznl\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.805003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804881 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.805003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.804960 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.805169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.805087 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.805304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.805273 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.805529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.805505 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.807053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.807029 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.807210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.807192 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.822283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.822256 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6"] Nov 02 21:32:31.826583 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:32:31.826557 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36377f3c_6f41_4ce3_8778_59f87630e3ab.slice/crio-39959a205ad95e537f6c61593a7eec7280c9f422879a4e0063391a502e27695c WatchSource:0}: Error finding container 39959a205ad95e537f6c61593a7eec7280c9f422879a4e0063391a502e27695c: Status 404 returned error can't find the container with id 39959a205ad95e537f6c61593a7eec7280c9f422879a4e0063391a502e27695c Nov 02 21:32:31.857683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.857658 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:31.858010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:31.857995 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwznl\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-kube-api-access-rwznl\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:32.154637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.154602 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:32.596933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.596902 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj"] Nov 02 21:32:32.601296 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:32:32.601265 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a7c99a_b621_4cdf_80eb_7eef3ab9320c.slice/crio-49ad9f93c42cb122425fb451e0442c7e725f1afbdbfe14501159d87d841927b6 WatchSource:0}: Error finding container 49ad9f93c42cb122425fb451e0442c7e725f1afbdbfe14501159d87d841927b6: Status 404 returned error can't find the container with id 49ad9f93c42cb122425fb451e0442c7e725f1afbdbfe14501159d87d841927b6 Nov 02 21:32:32.603823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.603794 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:32:32.603904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.603857 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:32:32.603904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.603884 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:32:32.632233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.632198 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" event={"ID":"36377f3c-6f41-4ce3-8778-59f87630e3ab","Type":"ContainerStarted","Data":"1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6"} Nov 02 21:32:32.632363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.632249 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" event={"ID":"36377f3c-6f41-4ce3-8778-59f87630e3ab","Type":"ContainerStarted","Data":"39959a205ad95e537f6c61593a7eec7280c9f422879a4e0063391a502e27695c"} Nov 02 21:32:32.634012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.633958 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" event={"ID":"81a7c99a-b621-4cdf-80eb-7eef3ab9320c","Type":"ContainerStarted","Data":"49ad9f93c42cb122425fb451e0442c7e725f1afbdbfe14501159d87d841927b6"} Nov 02 21:32:32.741272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:32.741219 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podStartSLOduration=2.74120592 podStartE2EDuration="2.74120592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:32.740304561 +0000 UTC m=+1182.295932898" watchObservedRunningTime="2026-11-02 21:32:32.74120592 +0000 UTC m=+1182.296834263" Nov 02 21:32:33.438752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:33.438723 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:33.643515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:33.643473 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" event={"ID":"81a7c99a-b621-4cdf-80eb-7eef3ab9320c","Type":"ContainerStarted","Data":"e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c"} Nov 02 21:32:34.155712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:34.155677 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:34.156947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:34.156920 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:32:34.157081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:34.156996 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 21:32:34.747243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:34.747192 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:35.155780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:35.155750 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:32:35.155960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:35.155801 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 21:32:36.155009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:36.154947 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:32:36.155602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:36.155040 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 21:32:37.155792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:37.155758 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:32:37.156285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:37.155819 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 21:32:38.155291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:38.155250 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:32:38.155500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:38.155302 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 21:32:39.155540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:39.155498 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:32:39.155935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:39.155579 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 21:32:39.400485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:39.400441 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:40.159680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:40.159652 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:40.160166 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:40.160152 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:40.160849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:40.160830 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:32:40.326037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:40.325942 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podStartSLOduration=9.325925748 podStartE2EDuration="9.325925748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:33.848257904 +0000 UTC m=+1183.403886240" watchObservedRunningTime="2026-11-02 21:32:40.325925748 +0000 UTC m=+1189.881554092" Nov 02 21:32:40.615494 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:32:40.615458 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:32:41.439031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:41.439001 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:41.451636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:41.451613 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:41.745395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:41.745317 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:32:43.887279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:43.887243 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:47.097101 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:47.097068 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:48.431002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:48.430952 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:50.766263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:50.766230 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:32:50.771267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:50.771241 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:32:50.781582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:50.781557 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:32:50.803371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:50.803345 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:32:50.823733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:50.823713 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:32:50.830960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:50.830924 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:32:52.968843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:52.968808 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:53.228033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.227935 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx"] Nov 02 21:32:53.231955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.231936 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.284845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.284815 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.285024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.285001 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznhh\" (UniqueName: \"kubernetes.io/projected/b85da4cf-900f-4647-a962-4140606a4a0b-kube-api-access-vznhh\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.285100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.285055 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.285589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.285564 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx"] Nov 02 21:32:53.326150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.326128 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:32:53.326150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.326131 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:32:53.326150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.326151 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:32:53.328445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.328423 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:32:53.393053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.393019 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.393179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.393153 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznhh\" (UniqueName: \"kubernetes.io/projected/b85da4cf-900f-4647-a962-4140606a4a0b-kube-api-access-vznhh\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.393255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.393203 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.395689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.395663 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.395982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.395947 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.448595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.448564 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznhh\" (UniqueName: \"kubernetes.io/projected/b85da4cf-900f-4647-a962-4140606a4a0b-kube-api-access-vznhh\") pod \"maas-api-revoke-sub-57c42232-5xlpx\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.545025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.544924 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:53.921336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:53.921305 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx"] Nov 02 21:32:53.925490 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:32:53.925462 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85da4cf_900f_4647_a962_4140606a4a0b.slice/crio-fad3902b001f13004969bcb2fee302df3e2272d337d3d615fab17ee9958e40ad WatchSource:0}: Error finding container fad3902b001f13004969bcb2fee302df3e2272d337d3d615fab17ee9958e40ad: Status 404 returned error can't find the container with id fad3902b001f13004969bcb2fee302df3e2272d337d3d615fab17ee9958e40ad Nov 02 21:32:54.844651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:54.844619 4441 generic.go:356] "Generic (PLEG): container finished" podID="b85da4cf-900f-4647-a962-4140606a4a0b" containerID="5e8eab0fe28a9bb5316cc6d8a2c29be26210d1c9b51da49bc3eb15ccfea83482" exitCode=0 Nov 02 21:32:54.845022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:54.844707 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" event={"ID":"b85da4cf-900f-4647-a962-4140606a4a0b","Type":"ContainerDied","Data":"5e8eab0fe28a9bb5316cc6d8a2c29be26210d1c9b51da49bc3eb15ccfea83482"} Nov 02 21:32:54.845022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:54.844754 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" event={"ID":"b85da4cf-900f-4647-a962-4140606a4a0b","Type":"ContainerStarted","Data":"fad3902b001f13004969bcb2fee302df3e2272d337d3d615fab17ee9958e40ad"} Nov 02 21:32:56.010159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.010135 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:56.125239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.125161 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-xks-ca\") pod \"b85da4cf-900f-4647-a962-4140606a4a0b\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " Nov 02 21:32:56.125401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.125271 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-openshift-ca\") pod \"b85da4cf-900f-4647-a962-4140606a4a0b\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " Nov 02 21:32:56.125401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.125308 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85da4cf-900f-4647-a962-4140606a4a0b-kube-api-access-vznhh\" (UniqueName: \"kubernetes.io/projected/b85da4cf-900f-4647-a962-4140606a4a0b-kube-api-access-vznhh\") pod \"b85da4cf-900f-4647-a962-4140606a4a0b\" (UID: \"b85da4cf-900f-4647-a962-4140606a4a0b\") " Nov 02 21:32:56.125681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.125653 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-openshift-ca" pod "b85da4cf-900f-4647-a962-4140606a4a0b" (UID: "b85da4cf-900f-4647-a962-4140606a4a0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:32:56.125860 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.125841 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:56.127766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.127741 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-xks-ca" pod "b85da4cf-900f-4647-a962-4140606a4a0b" (UID: "b85da4cf-900f-4647-a962-4140606a4a0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:32:56.127893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.127864 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85da4cf-900f-4647-a962-4140606a4a0b-kube-api-access-vznhh" pod "b85da4cf-900f-4647-a962-4140606a4a0b" (UID: "b85da4cf-900f-4647-a962-4140606a4a0b"). InnerVolumeSpecName "kube-api-access-vznhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:32:56.227449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.227415 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznhh\" (UniqueName: \"kubernetes.io/projected/b85da4cf-900f-4647-a962-4140606a4a0b-kube-api-access-vznhh\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:56.227449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.227444 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85da4cf-900f-4647-a962-4140606a4a0b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:56.864481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.864392 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" Nov 02 21:32:56.864481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.864429 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx" event={"ID":"b85da4cf-900f-4647-a962-4140606a4a0b","Type":"ContainerDied","Data":"fad3902b001f13004969bcb2fee302df3e2272d337d3d615fab17ee9958e40ad"} Nov 02 21:32:56.864763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:56.864485 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad3902b001f13004969bcb2fee302df3e2272d337d3d615fab17ee9958e40ad" Nov 02 21:32:57.569342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:32:57.569307 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:02.082948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:02.082850 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:06.838271 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:06.838234 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:08.050453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:08.050402 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc"] Nov 02 21:33:08.096524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:08.096486 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff1daeb-mx9kc"] Nov 02 21:33:08.584928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:08.584895 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69c6c9d-23d3-4be2-ad09-158a6cf8dbda" path="/var/lib/kubelet/pods/d69c6c9d-23d3-4be2-ad09-158a6cf8dbda/volumes" Nov 02 21:33:10.087819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:10.087781 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:10.615570 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:33:10.615535 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:33:11.526902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:11.526865 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:12.289010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.288960 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc"] Nov 02 21:33:12.289604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.289582 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b85da4cf-900f-4647-a962-4140606a4a0b" containerName="revoke-keys" Nov 02 21:33:12.289604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.289604 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85da4cf-900f-4647-a962-4140606a4a0b" containerName="revoke-keys" Nov 02 21:33:12.289809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.289796 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85da4cf-900f-4647-a962-4140606a4a0b" containerName="revoke-keys" Nov 02 21:33:12.293515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.293494 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.347403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.347371 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc"] Nov 02 21:33:12.366718 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.366686 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:12.387242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.387218 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"distinct-2-w2-4d9ea0-kserve-self-signed-certs\"" Nov 02 21:33:12.427711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.427678 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-dshm\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.427711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.427722 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94be10-a9ef-4259-b450-acd38f34d4e8-tls-certs\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.427920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.427792 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgm4\" (UniqueName: \"kubernetes.io/projected/ec94be10-a9ef-4259-b450-acd38f34d4e8-kube-api-access-wsgm4\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.427920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.427894 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-home\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.428074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.427990 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-tmp-dir\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.428131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.428082 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-model-cache\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529016 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94be10-a9ef-4259-b450-acd38f34d4e8-tls-certs\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529078 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgm4\" (UniqueName: \"kubernetes.io/projected/ec94be10-a9ef-4259-b450-acd38f34d4e8-kube-api-access-wsgm4\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529142 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-home\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529195 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-tmp-dir\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529258 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-model-cache\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529305 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-dshm\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529627 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-tmp-dir\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529626 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-home\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.529787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.529698 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-model-cache\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.531469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.531447 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-dshm\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.531592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.531568 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94be10-a9ef-4259-b450-acd38f34d4e8-tls-certs\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.583196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.583172 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgm4\" (UniqueName: \"kubernetes.io/projected/ec94be10-a9ef-4259-b450-acd38f34d4e8-kube-api-access-wsgm4\") pod \"distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.606106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.606084 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:12.953569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.953490 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6"] Nov 02 21:33:12.958264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.958242 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:12.990074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:12.990050 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc"] Nov 02 21:33:12.993909 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:12.993886 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec94be10_a9ef_4259_b450_acd38f34d4e8.slice/crio-b7b2280b107d75db445a4c1693f90f60e04601e7c6b370ee1f772a770375508c WatchSource:0}: Error finding container b7b2280b107d75db445a4c1693f90f60e04601e7c6b370ee1f772a770375508c: Status 404 returned error can't find the container with id b7b2280b107d75db445a4c1693f90f60e04601e7c6b370ee1f772a770375508c Nov 02 21:33:13.007648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.007623 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:13.008010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.007718 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:13.010752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.010725 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6"] Nov 02 21:33:13.024012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.023949 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" event={"ID":"ec94be10-a9ef-4259-b450-acd38f34d4e8","Type":"ContainerStarted","Data":"b7b2280b107d75db445a4c1693f90f60e04601e7c6b370ee1f772a770375508c"} Nov 02 21:33:13.032357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.032335 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:33:13.032466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.032367 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:13.136738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.136708 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.136900 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.136745 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrk2\" (UniqueName: \"kubernetes.io/projected/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-kube-api-access-kfrk2\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.137017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.137000 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.238336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.238249 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.238336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.238281 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrk2\" (UniqueName: \"kubernetes.io/projected/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-kube-api-access-kfrk2\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.238562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.238403 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.238955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.238930 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.240747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.240721 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.292438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.292408 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrk2\" (UniqueName: \"kubernetes.io/projected/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-kube-api-access-kfrk2\") pod \"maas-api-revoke-sub-fedc90f3-xrtf6\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.575654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.575623 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:13.959762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:13.959668 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6"] Nov 02 21:33:13.964612 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:13.964585 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dcd6d8_5266_4d2f_8b51_2577fbc431a5.slice/crio-f72d17b43eab44043ae064a441d065c56193bd40a5e7ec4d18550b29df14f92f WatchSource:0}: Error finding container f72d17b43eab44043ae064a441d065c56193bd40a5e7ec4d18550b29df14f92f: Status 404 returned error can't find the container with id f72d17b43eab44043ae064a441d065c56193bd40a5e7ec4d18550b29df14f92f Nov 02 21:33:14.032776 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:14.032745 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" event={"ID":"ec94be10-a9ef-4259-b450-acd38f34d4e8","Type":"ContainerStarted","Data":"ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef"} Nov 02 21:33:14.034282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:14.034263 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" event={"ID":"48dcd6d8-5266-4d2f-8b51-2577fbc431a5","Type":"ContainerStarted","Data":"f72d17b43eab44043ae064a441d065c56193bd40a5e7ec4d18550b29df14f92f"} Nov 02 21:33:14.145678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:14.145618 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podStartSLOduration=2.145604052 podStartE2EDuration="2.145604052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:14.140572162 +0000 UTC m=+1223.696200500" watchObservedRunningTime="2026-11-02 21:33:14.145604052 +0000 UTC m=+1223.701232389" Nov 02 21:33:14.606214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:14.606181 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:15.033293 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:33:15.033260 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dcd6d8_5266_4d2f_8b51_2577fbc431a5.slice/crio-conmon-5fb86449d54347e1b5ea765d30e1c86f85488d7bcb92873237179fc4faf6013f.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:33:15.043934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:15.043902 4441 generic.go:356] "Generic (PLEG): container finished" podID="48dcd6d8-5266-4d2f-8b51-2577fbc431a5" containerID="5fb86449d54347e1b5ea765d30e1c86f85488d7bcb92873237179fc4faf6013f" exitCode=0 Nov 02 21:33:15.044078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:15.043996 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" event={"ID":"48dcd6d8-5266-4d2f-8b51-2577fbc431a5","Type":"ContainerDied","Data":"5fb86449d54347e1b5ea765d30e1c86f85488d7bcb92873237179fc4faf6013f"} Nov 02 21:33:16.180448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.180411 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:16.214911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.214885 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:16.271754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.271723 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-kube-api-access-kfrk2\" (UniqueName: \"kubernetes.io/projected/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-kube-api-access-kfrk2\") pod \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " Nov 02 21:33:16.271925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.271781 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-xks-ca\") pod \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " Nov 02 21:33:16.272010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.271923 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-openshift-ca\") pod \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\" (UID: \"48dcd6d8-5266-4d2f-8b51-2577fbc431a5\") " Nov 02 21:33:16.272354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.272323 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-openshift-ca" pod "48dcd6d8-5266-4d2f-8b51-2577fbc431a5" (UID: "48dcd6d8-5266-4d2f-8b51-2577fbc431a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:16.274518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.274491 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-kube-api-access-kfrk2" pod "48dcd6d8-5266-4d2f-8b51-2577fbc431a5" (UID: "48dcd6d8-5266-4d2f-8b51-2577fbc431a5"). InnerVolumeSpecName "kube-api-access-kfrk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:16.275083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.275062 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-xks-ca" pod "48dcd6d8-5266-4d2f-8b51-2577fbc431a5" (UID: "48dcd6d8-5266-4d2f-8b51-2577fbc431a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:16.373824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.373790 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:16.373824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.373818 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfrk2\" (UniqueName: \"kubernetes.io/projected/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-kube-api-access-kfrk2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:16.373824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.373829 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48dcd6d8-5266-4d2f-8b51-2577fbc431a5-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:16.428395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.428366 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m"] Nov 02 21:33:16.428924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.428910 4441 cpu_manager.go:414] "containerMap: removing container" podUID="48dcd6d8-5266-4d2f-8b51-2577fbc431a5" containerName="revoke-keys" Nov 02 21:33:16.428996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.428926 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dcd6d8-5266-4d2f-8b51-2577fbc431a5" containerName="revoke-keys" Nov 02 21:33:16.429041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.429031 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dcd6d8-5266-4d2f-8b51-2577fbc431a5" containerName="revoke-keys" Nov 02 21:33:16.432627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.432606 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.475117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.475081 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.475362 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.475337 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28hvn\" (UniqueName: \"kubernetes.io/projected/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-kube-api-access-28hvn\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.475470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.475450 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.484754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.484727 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m"] Nov 02 21:33:16.577119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.577080 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28hvn\" (UniqueName: \"kubernetes.io/projected/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-kube-api-access-28hvn\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.577297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.577143 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.577297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.577226 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.578261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.578233 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.579615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.579590 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.630691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.630628 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28hvn\" (UniqueName: \"kubernetes.io/projected/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-kube-api-access-28hvn\") pod \"maas-api-revoke-sub-633c334c-tsf2m\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:16.754465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:16.754432 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:17.061682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:17.061651 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" Nov 02 21:33:17.061868 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:17.061680 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6" event={"ID":"48dcd6d8-5266-4d2f-8b51-2577fbc431a5","Type":"ContainerDied","Data":"f72d17b43eab44043ae064a441d065c56193bd40a5e7ec4d18550b29df14f92f"} Nov 02 21:33:17.061868 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:17.061753 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f72d17b43eab44043ae064a441d065c56193bd40a5e7ec4d18550b29df14f92f" Nov 02 21:33:17.136621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:17.136595 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m"] Nov 02 21:33:17.138496 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:17.138470 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222530ae_1b3b_4226_85eb_ae7f8bd6f3f8.slice/crio-6537194f335e543dec628f319c93ba848bf968e78bc14a91d6b31e0e29fd4c0d WatchSource:0}: Error finding container 6537194f335e543dec628f319c93ba848bf968e78bc14a91d6b31e0e29fd4c0d: Status 404 returned error can't find the container with id 6537194f335e543dec628f319c93ba848bf968e78bc14a91d6b31e0e29fd4c0d Nov 02 21:33:18.072946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:18.072910 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" event={"ID":"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8","Type":"ContainerStarted","Data":"f2a8f914ac74b15d90548f21e00c7d8253de928dd7dccba61c79e73b61265309"} Nov 02 21:33:18.073470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:18.072989 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" event={"ID":"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8","Type":"ContainerStarted","Data":"6537194f335e543dec628f319c93ba848bf968e78bc14a91d6b31e0e29fd4c0d"} Nov 02 21:33:18.182389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:18.182343 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" podStartSLOduration=2.1823298429999998 podStartE2EDuration="2.182329843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:18.180799997 +0000 UTC m=+1227.736428530" watchObservedRunningTime="2026-11-02 21:33:18.182329843 +0000 UTC m=+1227.737958179" Nov 02 21:33:19.082471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:19.082434 4441 generic.go:356] "Generic (PLEG): container finished" podID="222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" containerID="f2a8f914ac74b15d90548f21e00c7d8253de928dd7dccba61c79e73b61265309" exitCode=0 Nov 02 21:33:19.082958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:19.082539 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" event={"ID":"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8","Type":"ContainerDied","Data":"f2a8f914ac74b15d90548f21e00c7d8253de928dd7dccba61c79e73b61265309"} Nov 02 21:33:20.262020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.261994 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:20.324388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.324353 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-openshift-ca\") pod \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " Nov 02 21:33:20.324571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.324423 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-kube-api-access-28hvn\" (UniqueName: \"kubernetes.io/projected/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-kube-api-access-28hvn\") pod \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " Nov 02 21:33:20.324571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.324543 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-xks-ca\") pod \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\" (UID: \"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8\") " Nov 02 21:33:20.324674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.324646 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-openshift-ca" pod "222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" (UID: "222530ae-1b3b-4226-85eb-ae7f8bd6f3f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:20.325079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.325056 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:20.326575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.326548 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-xks-ca" pod "222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" (UID: "222530ae-1b3b-4226-85eb-ae7f8bd6f3f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:20.326662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.326612 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-kube-api-access-28hvn" pod "222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" (UID: "222530ae-1b3b-4226-85eb-ae7f8bd6f3f8"). InnerVolumeSpecName "kube-api-access-28hvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:20.426201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.426112 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28hvn\" (UniqueName: \"kubernetes.io/projected/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-kube-api-access-28hvn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:20.426201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.426140 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:20.792469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:20.792377 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:21.104898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:21.104863 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" Nov 02 21:33:21.105097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:21.104892 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m" event={"ID":"222530ae-1b3b-4226-85eb-ae7f8bd6f3f8","Type":"ContainerDied","Data":"6537194f335e543dec628f319c93ba848bf968e78bc14a91d6b31e0e29fd4c0d"} Nov 02 21:33:21.105097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:21.104994 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6537194f335e543dec628f319c93ba848bf968e78bc14a91d6b31e0e29fd4c0d" Nov 02 21:33:22.606907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:22.606876 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:22.619609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:22.619585 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:23.134669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:23.134639 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:25.090544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:25.090476 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f"] Nov 02 21:33:25.132543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:25.132477 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn"] Nov 02 21:33:25.255234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:25.255193 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376d2436-qmr2f"] Nov 02 21:33:25.426292 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:25.426202 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e6b02b-5lgrn"] Nov 02 21:33:25.720044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:25.719101 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:26.429202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:26.429170 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:26.585735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:26.585692 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b5cee1-49ba-4747-bf53-be0892a2a431" path="/var/lib/kubelet/pods/38b5cee1-49ba-4747-bf53-be0892a2a431/volumes" Nov 02 21:33:26.586243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:26.586219 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2f4ac1-039c-4ce3-be32-abb3af240960" path="/var/lib/kubelet/pods/3c2f4ac1-039c-4ce3-be32-abb3af240960/volumes" Nov 02 21:33:27.278537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.278504 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t"] Nov 02 21:33:27.279069 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.279055 4441 cpu_manager.go:414] "containerMap: removing container" podUID="222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" containerName="revoke-keys" Nov 02 21:33:27.279123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.279070 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" containerName="revoke-keys" Nov 02 21:33:27.279174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.279166 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" containerName="revoke-keys" Nov 02 21:33:27.283924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.283904 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.335329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.335306 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t"] Nov 02 21:33:27.363470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.363430 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.363606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.363504 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.363606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.363546 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkqc\" (UniqueName: \"kubernetes.io/projected/2b8d1813-c9b3-4811-b133-d3c2952822fb-kube-api-access-pfkqc\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.376859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.376834 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:33:27.376859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.376834 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:27.377053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.376873 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:27.377155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.377128 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:27.464931 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.464888 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.465349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.464952 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkqc\" (UniqueName: \"kubernetes.io/projected/2b8d1813-c9b3-4811-b133-d3c2952822fb-kube-api-access-pfkqc\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.465349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.465137 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.465651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.465615 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.467617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.467594 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.519877 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.519851 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkqc\" (UniqueName: \"kubernetes.io/projected/2b8d1813-c9b3-4811-b133-d3c2952822fb-kube-api-access-pfkqc\") pod \"maas-api-revoke-sub-24ddfffc-q5p9t\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.597527 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.597501 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:27.983741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:27.983663 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t"] Nov 02 21:33:27.988426 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:27.988395 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8d1813_c9b3_4811_b133_d3c2952822fb.slice/crio-e6b140c30c9e2149cff7f0c8d496c58883a786ef94a1888f77b45a51e8bff7bd WatchSource:0}: Error finding container e6b140c30c9e2149cff7f0c8d496c58883a786ef94a1888f77b45a51e8bff7bd: Status 404 returned error can't find the container with id e6b140c30c9e2149cff7f0c8d496c58883a786ef94a1888f77b45a51e8bff7bd Nov 02 21:33:28.400589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.400556 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" event={"ID":"2b8d1813-c9b3-4811-b133-d3c2952822fb","Type":"ContainerStarted","Data":"7d0d22f215f133403451db1acce65cecd5a6915a6cf4bc48c68dfcea96e0b941"} Nov 02 21:33:28.400763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.400618 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" event={"ID":"2b8d1813-c9b3-4811-b133-d3c2952822fb","Type":"ContainerStarted","Data":"e6b140c30c9e2149cff7f0c8d496c58883a786ef94a1888f77b45a51e8bff7bd"} Nov 02 21:33:28.512021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.511938 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" podStartSLOduration=1.511920283 podStartE2EDuration="1.511920283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:28.507515368 +0000 UTC m=+1238.063143706" watchObservedRunningTime="2026-11-02 21:33:28.511920283 +0000 UTC m=+1238.067548623" Nov 02 21:33:28.601516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.601483 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd"] Nov 02 21:33:28.605682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.605667 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.658903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.658877 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd"] Nov 02 21:33:28.681051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.681022 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvsk\" (UniqueName: \"kubernetes.io/projected/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-kube-api-access-xkvsk\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.681236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.681216 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.681319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.681250 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.782717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.782686 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvsk\" (UniqueName: \"kubernetes.io/projected/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-kube-api-access-xkvsk\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.782847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.782795 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.782847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.782820 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.783654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.783628 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.785298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.785275 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.836909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.836883 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvsk\" (UniqueName: \"kubernetes.io/projected/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-kube-api-access-xkvsk\") pod \"maas-api-revoke-sub-5f0cdac9-52knd\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:28.921862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:28.921793 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:29.301815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:29.301734 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd"] Nov 02 21:33:29.307194 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:29.307161 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7ecc8e_f35b_4d79_ba0f_e62e6bded104.slice/crio-067a1ed24e10da4e9580cb9ebdf0bb1fc4e85e86727681bf26ea606659ce46e0 WatchSource:0}: Error finding container 067a1ed24e10da4e9580cb9ebdf0bb1fc4e85e86727681bf26ea606659ce46e0: Status 404 returned error can't find the container with id 067a1ed24e10da4e9580cb9ebdf0bb1fc4e85e86727681bf26ea606659ce46e0 Nov 02 21:33:29.412741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:29.412699 4441 generic.go:356] "Generic (PLEG): container finished" podID="2b8d1813-c9b3-4811-b133-d3c2952822fb" containerID="7d0d22f215f133403451db1acce65cecd5a6915a6cf4bc48c68dfcea96e0b941" exitCode=0 Nov 02 21:33:29.412935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:29.412788 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" event={"ID":"2b8d1813-c9b3-4811-b133-d3c2952822fb","Type":"ContainerDied","Data":"7d0d22f215f133403451db1acce65cecd5a6915a6cf4bc48c68dfcea96e0b941"} Nov 02 21:33:29.415177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:29.415151 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" event={"ID":"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104","Type":"ContainerStarted","Data":"067a1ed24e10da4e9580cb9ebdf0bb1fc4e85e86727681bf26ea606659ce46e0"} Nov 02 21:33:30.371513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.371488 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:30.426168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.426129 4441 generic.go:356] "Generic (PLEG): container finished" podID="7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" containerID="0a61bfb31f564c0495b26be84da914acfb770c76b91cc93a27b799f38279253b" exitCode=0 Nov 02 21:33:30.426364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.426218 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" event={"ID":"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104","Type":"ContainerDied","Data":"0a61bfb31f564c0495b26be84da914acfb770c76b91cc93a27b799f38279253b"} Nov 02 21:33:30.600578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.600509 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:30.710031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.709998 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8d1813-c9b3-4811-b133-d3c2952822fb-kube-api-access-pfkqc\" (UniqueName: \"kubernetes.io/projected/2b8d1813-c9b3-4811-b133-d3c2952822fb-kube-api-access-pfkqc\") pod \"2b8d1813-c9b3-4811-b133-d3c2952822fb\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " Nov 02 21:33:30.710206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.710168 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-openshift-ca\") pod \"2b8d1813-c9b3-4811-b133-d3c2952822fb\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " Nov 02 21:33:30.710259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.710205 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-xks-ca\") pod \"2b8d1813-c9b3-4811-b133-d3c2952822fb\" (UID: \"2b8d1813-c9b3-4811-b133-d3c2952822fb\") " Nov 02 21:33:30.710613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.710582 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-openshift-ca" pod "2b8d1813-c9b3-4811-b133-d3c2952822fb" (UID: "2b8d1813-c9b3-4811-b133-d3c2952822fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:30.710789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.710758 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:30.712689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.712648 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8d1813-c9b3-4811-b133-d3c2952822fb-kube-api-access-pfkqc" pod "2b8d1813-c9b3-4811-b133-d3c2952822fb" (UID: "2b8d1813-c9b3-4811-b133-d3c2952822fb"). InnerVolumeSpecName "kube-api-access-pfkqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:30.712792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.712700 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-xks-ca" pod "2b8d1813-c9b3-4811-b133-d3c2952822fb" (UID: "2b8d1813-c9b3-4811-b133-d3c2952822fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:30.812530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.812484 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8d1813-c9b3-4811-b133-d3c2952822fb-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:30.812530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:30.812529 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkqc\" (UniqueName: \"kubernetes.io/projected/2b8d1813-c9b3-4811-b133-d3c2952822fb-kube-api-access-pfkqc\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:31.436568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.436533 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" Nov 02 21:33:31.436568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.436557 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t" event={"ID":"2b8d1813-c9b3-4811-b133-d3c2952822fb","Type":"ContainerDied","Data":"e6b140c30c9e2149cff7f0c8d496c58883a786ef94a1888f77b45a51e8bff7bd"} Nov 02 21:33:31.437083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.436610 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b140c30c9e2149cff7f0c8d496c58883a786ef94a1888f77b45a51e8bff7bd" Nov 02 21:33:31.591705 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.591676 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:31.723345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.723255 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-xks-ca\") pod \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " Nov 02 21:33:31.723510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.723401 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-kube-api-access-xkvsk\" (UniqueName: \"kubernetes.io/projected/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-kube-api-access-xkvsk\") pod \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " Nov 02 21:33:31.723510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.723421 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-openshift-ca\") pod \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\" (UID: \"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104\") " Nov 02 21:33:31.723902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.723860 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-openshift-ca" pod "7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" (UID: "7b7ecc8e-f35b-4d79-ba0f-e62e6bded104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:31.725348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.725321 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-xks-ca" pod "7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" (UID: "7b7ecc8e-f35b-4d79-ba0f-e62e6bded104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:31.725465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.725446 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-kube-api-access-xkvsk" pod "7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" (UID: "7b7ecc8e-f35b-4d79-ba0f-e62e6bded104"). InnerVolumeSpecName "kube-api-access-xkvsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:31.825466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.825435 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:31.825466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.825465 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvsk\" (UniqueName: \"kubernetes.io/projected/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-kube-api-access-xkvsk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:31.825637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.825477 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:31.999755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:31.999679 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2"] Nov 02 21:33:32.000337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.000315 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2b8d1813-c9b3-4811-b133-d3c2952822fb" containerName="revoke-keys" Nov 02 21:33:32.000453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.000337 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8d1813-c9b3-4811-b133-d3c2952822fb" containerName="revoke-keys" Nov 02 21:33:32.000453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.000351 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" containerName="revoke-keys" Nov 02 21:33:32.000453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.000358 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" containerName="revoke-keys" Nov 02 21:33:32.000624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.000508 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8d1813-c9b3-4811-b133-d3c2952822fb" containerName="revoke-keys" Nov 02 21:33:32.000624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.000524 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" containerName="revoke-keys" Nov 02 21:33:32.004448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.004428 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.099133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.099101 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2"] Nov 02 21:33:32.129209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129181 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129300 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129332 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129390 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129419 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129441 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129525 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th5l\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-kube-api-access-5th5l\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129632 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07c656b-5fbf-4a92-a8f3-214769910deb-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.129678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.129670 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230520 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230562 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230584 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230609 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230629 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230647 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th5l\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-kube-api-access-5th5l\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230722 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07c656b-5fbf-4a92-a8f3-214769910deb-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.230793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230751 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.231141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.230894 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.231141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.231054 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.231248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.231143 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.231248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.231191 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.231333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.231269 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.231649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.231625 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07c656b-5fbf-4a92-a8f3-214769910deb-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.233051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.233025 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.233252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.233232 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.284430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.284366 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.284430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.284429 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th5l\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-kube-api-access-5th5l\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.319013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.318991 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:32.449053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.449016 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" event={"ID":"7b7ecc8e-f35b-4d79-ba0f-e62e6bded104","Type":"ContainerDied","Data":"067a1ed24e10da4e9580cb9ebdf0bb1fc4e85e86727681bf26ea606659ce46e0"} Nov 02 21:33:32.449474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.449080 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="067a1ed24e10da4e9580cb9ebdf0bb1fc4e85e86727681bf26ea606659ce46e0" Nov 02 21:33:32.449474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.449035 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd" Nov 02 21:33:32.758499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.758468 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2"] Nov 02 21:33:32.762027 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:32.762003 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07c656b_5fbf_4a92_a8f3_214769910deb.slice/crio-a2fe4bbd26315748ff6819d461a8d8e0e94b299cfff19a2d8c495abde5fc5541 WatchSource:0}: Error finding container a2fe4bbd26315748ff6819d461a8d8e0e94b299cfff19a2d8c495abde5fc5541: Status 404 returned error can't find the container with id a2fe4bbd26315748ff6819d461a8d8e0e94b299cfff19a2d8c495abde5fc5541 Nov 02 21:33:32.764400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.764364 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:33:32.764496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.764438 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:33:32.764496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:32.764466 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:33:33.458424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:33.458386 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" event={"ID":"e07c656b-5fbf-4a92-a8f3-214769910deb","Type":"ContainerStarted","Data":"4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09"} Nov 02 21:33:33.458824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:33.458444 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" event={"ID":"e07c656b-5fbf-4a92-a8f3-214769910deb","Type":"ContainerStarted","Data":"a2fe4bbd26315748ff6819d461a8d8e0e94b299cfff19a2d8c495abde5fc5541"} Nov 02 21:33:33.583992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:33.574776 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podStartSLOduration=2.574757115 podStartE2EDuration="2.574757115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:33.568288127 +0000 UTC m=+1243.123916464" watchObservedRunningTime="2026-11-02 21:33:33.574757115 +0000 UTC m=+1243.130385452" Nov 02 21:33:33.953316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:33.953282 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:34.326995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:34.325157 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:34.326995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:34.325287 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 21:33:34.326995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:34.325330 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 21:33:35.092448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:35.092405 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:35.319720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:35.319684 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 21:33:35.319893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:35.319751 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 21:33:36.320413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:36.320377 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 21:33:36.320884 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:36.320448 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 21:33:37.320056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:37.320010 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 21:33:37.320270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:37.320078 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 21:33:38.319771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:38.319734 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 21:33:38.320190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:38.319797 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 21:33:39.319588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:39.319550 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 21:33:39.319771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:39.319619 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 21:33:39.591136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:39.591051 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:40.048932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:40.048852 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5"] Nov 02 21:33:40.095687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:40.095655 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a408e0-757j5"] Nov 02 21:33:40.323772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:40.323739 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:40.324219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:40.324203 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:40.324838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:40.324818 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:33:40.590182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:40.589891 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49dfb678-11b2-4620-832b-7e1db1b871ab" path="/var/lib/kubelet/pods/49dfb678-11b2-4620-832b-7e1db1b871ab/volumes" Nov 02 21:33:40.622613 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:33:40.622582 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:33:42.463959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.463927 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6"] Nov 02 21:33:42.468179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.468163 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.528703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.528670 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6"] Nov 02 21:33:42.546416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.546389 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:42.566264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.566242 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"distinct-2-w5-ef6b43-kserve-self-signed-certs\"" Nov 02 21:33:42.591022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.590994 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmdj\" (UniqueName: \"kubernetes.io/projected/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-kube-api-access-dpmdj\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.591148 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.591124 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-home\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.591188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.591170 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-model-cache\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.591235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.591206 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-dshm\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.591369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.591349 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tmp-dir\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.591453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.591438 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tls-certs\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.692804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.692766 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-home\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.692804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.692808 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-model-cache\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.693078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.692831 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-dshm\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.693078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.692997 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tmp-dir\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.693189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.693134 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tls-certs\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.693248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.693224 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-home\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.693306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.693227 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmdj\" (UniqueName: \"kubernetes.io/projected/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-kube-api-access-dpmdj\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.693306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.693267 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-model-cache\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.693450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.693426 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tmp-dir\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.695238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.695209 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-dshm\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.695471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.695455 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tls-certs\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.747324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.747239 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmdj\" (UniqueName: \"kubernetes.io/projected/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-kube-api-access-dpmdj\") pod \"distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:42.783192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:42.783158 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:43.180284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:43.180251 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6"] Nov 02 21:33:43.184150 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:43.184119 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d2e45a9_0f78_485e_9f18_a3e376ca1e8c.slice/crio-a57f1abe5a5078f15aec1d1e0fa9eb4a1d0abc03370005fa692826e40e170946 WatchSource:0}: Error finding container a57f1abe5a5078f15aec1d1e0fa9eb4a1d0abc03370005fa692826e40e170946: Status 404 returned error can't find the container with id a57f1abe5a5078f15aec1d1e0fa9eb4a1d0abc03370005fa692826e40e170946 Nov 02 21:33:43.566279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:43.566246 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" event={"ID":"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c","Type":"ContainerStarted","Data":"e26868832556b48b42ccdfeabe1de4b2d771260194f60d806eb56bbd791011b4"} Nov 02 21:33:43.566681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:43.566292 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" event={"ID":"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c","Type":"ContainerStarted","Data":"a57f1abe5a5078f15aec1d1e0fa9eb4a1d0abc03370005fa692826e40e170946"} Nov 02 21:33:43.674616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:43.674563 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podStartSLOduration=1.674549967 podStartE2EDuration="1.674549967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:43.674131748 +0000 UTC m=+1253.229760089" watchObservedRunningTime="2026-11-02 21:33:43.674549967 +0000 UTC m=+1253.230178321" Nov 02 21:33:43.784019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:43.783959 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:44.107719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:44.107655 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:45.048178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.048142 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv"] Nov 02 21:33:45.094566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.094538 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dff1ca0-5kxqv"] Nov 02 21:33:45.226050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.226015 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr"] Nov 02 21:33:45.230280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.230261 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.266484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.266450 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wj4p\" (UniqueName: \"kubernetes.io/projected/a23e46eb-0250-4933-9de6-74cd41b3b94a-kube-api-access-6wj4p\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.266613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.266508 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.266781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.266765 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.279910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.279882 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:45.279910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.279888 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:45.280128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.279890 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:45.280128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.279957 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:33:45.281765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.281744 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr"] Nov 02 21:33:45.368539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.368508 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.368714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.368582 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wj4p\" (UniqueName: \"kubernetes.io/projected/a23e46eb-0250-4933-9de6-74cd41b3b94a-kube-api-access-6wj4p\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.368714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.368614 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.369278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.369257 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.370880 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.370856 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.422363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.422335 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wj4p\" (UniqueName: \"kubernetes.io/projected/a23e46eb-0250-4933-9de6-74cd41b3b94a-kube-api-access-6wj4p\") pod \"maas-api-revoke-sub-25467252-jg7lr\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.544889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.544846 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:45.935846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:45.935775 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr"] Nov 02 21:33:45.940677 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:45.940649 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23e46eb_0250_4933_9de6_74cd41b3b94a.slice/crio-81d3922c49c1819468cf43dbaa33f9a7159d259e6addbbb913b79c067fe614bf WatchSource:0}: Error finding container 81d3922c49c1819468cf43dbaa33f9a7159d259e6addbbb913b79c067fe614bf: Status 404 returned error can't find the container with id 81d3922c49c1819468cf43dbaa33f9a7159d259e6addbbb913b79c067fe614bf Nov 02 21:33:46.587324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:46.587296 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d01c62f-bba3-476e-98c3-603e69e6ecf3" path="/var/lib/kubelet/pods/0d01c62f-bba3-476e-98c3-603e69e6ecf3/volumes" Nov 02 21:33:46.598668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:46.598641 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" event={"ID":"a23e46eb-0250-4933-9de6-74cd41b3b94a","Type":"ContainerStarted","Data":"6fad5b304d22266db8190f5be372f279ef1098b45eb5f02fa82e10b70bcc7491"} Nov 02 21:33:46.598823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:46.598681 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" event={"ID":"a23e46eb-0250-4933-9de6-74cd41b3b94a","Type":"ContainerStarted","Data":"81d3922c49c1819468cf43dbaa33f9a7159d259e6addbbb913b79c067fe614bf"} Nov 02 21:33:46.707756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:46.707703 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" podStartSLOduration=1.707690999 podStartE2EDuration="1.707690999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:46.704846572 +0000 UTC m=+1256.260474911" watchObservedRunningTime="2026-11-02 21:33:46.707690999 +0000 UTC m=+1256.263319336" Nov 02 21:33:47.453810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:47.453779 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:47.609866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:47.609826 4441 generic.go:356] "Generic (PLEG): container finished" podID="a23e46eb-0250-4933-9de6-74cd41b3b94a" containerID="6fad5b304d22266db8190f5be372f279ef1098b45eb5f02fa82e10b70bcc7491" exitCode=0 Nov 02 21:33:47.610375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:47.609919 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" event={"ID":"a23e46eb-0250-4933-9de6-74cd41b3b94a","Type":"ContainerDied","Data":"6fad5b304d22266db8190f5be372f279ef1098b45eb5f02fa82e10b70bcc7491"} Nov 02 21:33:48.565545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.565515 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc"] Nov 02 21:33:48.569675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.569652 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609151 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609268 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609322 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9pj\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-kube-api-access-vw9pj\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609365 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609501 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609709 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609692 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609788 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609856 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609816 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51771762-996a-41a8-b5a1-8dabdd4c8524-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.609909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.609871 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.673685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.673654 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc"] Nov 02 21:33:48.703303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.703238 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711407 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711516 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711545 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51771762-996a-41a8-b5a1-8dabdd4c8524-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711576 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711615 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711674 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711728 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9pj\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-kube-api-access-vw9pj\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711764 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.711792 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.712025 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.712278 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.712402 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51771762-996a-41a8-b5a1-8dabdd4c8524-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.712670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.712510 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.713435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.712683 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.715612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.715593 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.715835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.715809 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.779045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.779007 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9pj\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-kube-api-access-vw9pj\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.781405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.781372 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.844560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.844537 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:48.883914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.883888 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:48.914855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.914824 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-openshift-ca\") pod \"a23e46eb-0250-4933-9de6-74cd41b3b94a\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " Nov 02 21:33:48.914993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.914939 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23e46eb-0250-4933-9de6-74cd41b3b94a-kube-api-access-6wj4p\" (UniqueName: \"kubernetes.io/projected/a23e46eb-0250-4933-9de6-74cd41b3b94a-kube-api-access-6wj4p\") pod \"a23e46eb-0250-4933-9de6-74cd41b3b94a\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " Nov 02 21:33:48.915068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.915031 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-xks-ca\") pod \"a23e46eb-0250-4933-9de6-74cd41b3b94a\" (UID: \"a23e46eb-0250-4933-9de6-74cd41b3b94a\") " Nov 02 21:33:48.915480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.915452 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-openshift-ca" pod "a23e46eb-0250-4933-9de6-74cd41b3b94a" (UID: "a23e46eb-0250-4933-9de6-74cd41b3b94a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:48.915635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.915616 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:48.917788 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.917763 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23e46eb-0250-4933-9de6-74cd41b3b94a-kube-api-access-6wj4p" pod "a23e46eb-0250-4933-9de6-74cd41b3b94a" (UID: "a23e46eb-0250-4933-9de6-74cd41b3b94a"). InnerVolumeSpecName "kube-api-access-6wj4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:48.917869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:48.917840 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-xks-ca" pod "a23e46eb-0250-4933-9de6-74cd41b3b94a" (UID: "a23e46eb-0250-4933-9de6-74cd41b3b94a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:49.017573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.017538 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e46eb-0250-4933-9de6-74cd41b3b94a-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:49.017573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.017570 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wj4p\" (UniqueName: \"kubernetes.io/projected/a23e46eb-0250-4933-9de6-74cd41b3b94a-kube-api-access-6wj4p\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:49.319277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.319247 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc"] Nov 02 21:33:49.322689 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:49.322660 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51771762_996a_41a8_b5a1_8dabdd4c8524.slice/crio-0f61db41763a11f5e81d85a956e3f7442c99df4e6f8d71bd35bfd676a7597926 WatchSource:0}: Error finding container 0f61db41763a11f5e81d85a956e3f7442c99df4e6f8d71bd35bfd676a7597926: Status 404 returned error can't find the container with id 0f61db41763a11f5e81d85a956e3f7442c99df4e6f8d71bd35bfd676a7597926 Nov 02 21:33:49.325033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.325003 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:33:49.325155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.325067 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:33:49.325155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.325100 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:33:49.633323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.633242 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" Nov 02 21:33:49.633775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.633330 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr" event={"ID":"a23e46eb-0250-4933-9de6-74cd41b3b94a","Type":"ContainerDied","Data":"81d3922c49c1819468cf43dbaa33f9a7159d259e6addbbb913b79c067fe614bf"} Nov 02 21:33:49.633775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.633417 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d3922c49c1819468cf43dbaa33f9a7159d259e6addbbb913b79c067fe614bf" Nov 02 21:33:49.643348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.643043 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" event={"ID":"51771762-996a-41a8-b5a1-8dabdd4c8524","Type":"ContainerStarted","Data":"b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e"} Nov 02 21:33:49.643348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.643118 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" event={"ID":"51771762-996a-41a8-b5a1-8dabdd4c8524","Type":"ContainerStarted","Data":"0f61db41763a11f5e81d85a956e3f7442c99df4e6f8d71bd35bfd676a7597926"} Nov 02 21:33:49.809863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.809793 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podStartSLOduration=1.809773523 podStartE2EDuration="1.809773523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:49.805773228 +0000 UTC m=+1259.361401565" watchObservedRunningTime="2026-11-02 21:33:49.809773523 +0000 UTC m=+1259.365401860" Nov 02 21:33:49.885144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:49.885058 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:33:50.732195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.732160 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png"] Nov 02 21:33:50.732763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.732736 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a23e46eb-0250-4933-9de6-74cd41b3b94a" containerName="revoke-keys" Nov 02 21:33:50.732763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.732752 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23e46eb-0250-4933-9de6-74cd41b3b94a" containerName="revoke-keys" Nov 02 21:33:50.732873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.732857 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23e46eb-0250-4933-9de6-74cd41b3b94a" containerName="revoke-keys" Nov 02 21:33:50.737422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.737401 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.806337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.806300 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png"] Nov 02 21:33:50.832276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.832243 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:50.832445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.832329 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:50.832518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.832260 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:50.839406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.839383 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.839571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.839410 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.839571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.839455 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq675\" (UniqueName: \"kubernetes.io/projected/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-kube-api-access-tq675\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.850865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.850848 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:33:50.905806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.905751 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:33:50.905990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.905824 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 21:33:50.940703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.940666 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.940703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.940704 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.940884 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.940753 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq675\" (UniqueName: \"kubernetes.io/projected/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-kube-api-access-tq675\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.941359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.941333 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:50.943110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:50.943085 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:51.001035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:51.000931 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq675\" (UniqueName: \"kubernetes.io/projected/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-kube-api-access-tq675\") pod \"maas-api-revoke-sub-e7ed3418-v7png\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:51.053559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:51.053528 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:51.506254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:51.506210 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png"] Nov 02 21:33:51.511015 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:33:51.510949 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5809dc_15a3_4789_9fb8_9b6e398d57cf.slice/crio-dc5647c1951f0158d3bae5351dfe94ab4034f0b0632b8c9e44124738dca45118 WatchSource:0}: Error finding container dc5647c1951f0158d3bae5351dfe94ab4034f0b0632b8c9e44124738dca45118: Status 404 returned error can't find the container with id dc5647c1951f0158d3bae5351dfe94ab4034f0b0632b8c9e44124738dca45118 Nov 02 21:33:51.661912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:51.661880 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" event={"ID":"7d5809dc-15a3-4789-9fb8-9b6e398d57cf","Type":"ContainerStarted","Data":"dc5647c1951f0158d3bae5351dfe94ab4034f0b0632b8c9e44124738dca45118"} Nov 02 21:33:51.893994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:51.886226 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:33:51.893994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:51.886285 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 21:33:52.690992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:52.690619 4441 generic.go:356] "Generic (PLEG): container finished" podID="7d5809dc-15a3-4789-9fb8-9b6e398d57cf" containerID="19a6bc11df362fc8de44c43774aa9fa6d34ec1c595f98b798905e2546323a312" exitCode=0 Nov 02 21:33:52.694992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:52.691429 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" event={"ID":"7d5809dc-15a3-4789-9fb8-9b6e398d57cf","Type":"ContainerDied","Data":"19a6bc11df362fc8de44c43774aa9fa6d34ec1c595f98b798905e2546323a312"} Nov 02 21:33:52.802790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:52.800952 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:52.815784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:52.815755 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:52.892431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:52.892048 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:33:52.892431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:52.892099 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 21:33:53.323315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:53.323278 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:53.720191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:53.720162 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:53.914689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:53.914654 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:33:53.921103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:53.921064 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 21:33:53.967533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:53.967468 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:54.119545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.118463 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-xks-ca\") pod \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " Nov 02 21:33:54.119545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.118611 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-kube-api-access-tq675\" (UniqueName: \"kubernetes.io/projected/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-kube-api-access-tq675\") pod \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " Nov 02 21:33:54.119545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.118718 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-openshift-ca\") pod \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\" (UID: \"7d5809dc-15a3-4789-9fb8-9b6e398d57cf\") " Nov 02 21:33:54.120018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.119993 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-openshift-ca" pod "7d5809dc-15a3-4789-9fb8-9b6e398d57cf" (UID: "7d5809dc-15a3-4789-9fb8-9b6e398d57cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:54.125097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.124503 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-xks-ca" pod "7d5809dc-15a3-4789-9fb8-9b6e398d57cf" (UID: "7d5809dc-15a3-4789-9fb8-9b6e398d57cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:54.132563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.132539 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-kube-api-access-tq675" pod "7d5809dc-15a3-4789-9fb8-9b6e398d57cf" (UID: "7d5809dc-15a3-4789-9fb8-9b6e398d57cf"). InnerVolumeSpecName "kube-api-access-tq675". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:54.220814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.220717 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:54.220814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.220748 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq675\" (UniqueName: \"kubernetes.io/projected/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-kube-api-access-tq675\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:54.220814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.220761 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5809dc-15a3-4789-9fb8-9b6e398d57cf-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:54.716097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.716064 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" Nov 02 21:33:54.716478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.716094 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png" event={"ID":"7d5809dc-15a3-4789-9fb8-9b6e398d57cf","Type":"ContainerDied","Data":"dc5647c1951f0158d3bae5351dfe94ab4034f0b0632b8c9e44124738dca45118"} Nov 02 21:33:54.716478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.716144 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5647c1951f0158d3bae5351dfe94ab4034f0b0632b8c9e44124738dca45118" Nov 02 21:33:54.888049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.887001 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:33:54.888049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:54.887058 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 21:33:55.597018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:55.596950 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:55.716670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:55.716630 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:55.889569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:55.889474 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:33:55.889867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:55.889827 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 21:33:56.104993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:56.103010 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6"] Nov 02 21:33:56.125536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:56.122424 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8963e511-f5cg6"] Nov 02 21:33:56.638992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:56.633295 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e84eff7-4414-406d-9026-f029e62d977a" path="/var/lib/kubelet/pods/8e84eff7-4414-406d-9026-f029e62d977a/volumes" Nov 02 21:33:56.889400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:56.889296 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:33:56.890010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:56.889962 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 21:33:57.401170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.401110 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.440173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.440110 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.523487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.523442 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 21:33:57.644142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.644081 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.707890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.707793 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": context deadline exceeded" Nov 02 21:33:57.740085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.740041 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.869110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.869064 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.869659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:57.869629 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:33:58.060990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.059057 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.060990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.060128 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.060990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.060202 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.118993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.116214 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.128694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.128638 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Nov 02 21:33:58.229732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.229687 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.249090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.249046 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.450188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.450084 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.450994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.450769 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:33:58.478990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.477061 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.533266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.533221 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.533851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.533827 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:33:58.658135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.651208 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:33:58.736038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.727035 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.736038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.735917 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:33:58.736038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.736007 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:33:58.736430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.736044 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:58.743194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.740899 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.743194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.741269 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:33:58.884993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.869754 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.961988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.958882 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:33:59.004049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:58.971174 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:33:59.007466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.006624 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:33:59.033327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.033280 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:33:59.033689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.033649 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:59.061130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.060795 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 21:33:59.063015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.061748 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:33:59.063015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.062626 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:59.063015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.062851 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:33:59.149007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.140041 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Nov 02 21:33:59.149007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.142216 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:59.202084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.179690 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:59.252004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.244601 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Nov 02 21:33:59.266897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.265408 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:33:59.309994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.308458 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:33:59.309994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.309270 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:33:59.379488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.375613 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:33:59.395430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.394501 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:33:59.455866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.455823 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:33:59.485763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.483405 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:33:59.486995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.486647 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:33:59.488916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.488051 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:33:59.488916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:33:59.488100 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:34:00.077141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.077103 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:34:00.077730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.077697 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:00.810264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.810234 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:34:00.810589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.810563 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:34:00.823869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.822808 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:34:00.838001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.837952 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp"] Nov 02 21:34:00.838861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.838829 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7d5809dc-15a3-4789-9fb8-9b6e398d57cf" containerName="revoke-keys" Nov 02 21:34:00.838981 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.838864 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5809dc-15a3-4789-9fb8-9b6e398d57cf" containerName="revoke-keys" Nov 02 21:34:00.843160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.843135 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5809dc-15a3-4789-9fb8-9b6e398d57cf" containerName="revoke-keys" Nov 02 21:34:00.848896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.848864 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:00.890164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.890129 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp"] Nov 02 21:34:00.903365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.903318 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:34:00.903670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.903646 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:34:00.903784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.903703 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:34:00.903848 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.903810 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:34:00.982008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.981952 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:00.982174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.982082 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9sm2\" (UniqueName: \"kubernetes.io/projected/afc759d7-bd00-4e59-b3cf-3486fba4e78b-kube-api-access-n9sm2\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:00.982174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:00.982123 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.083836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.083749 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.084299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.083904 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9sm2\" (UniqueName: \"kubernetes.io/projected/afc759d7-bd00-4e59-b3cf-3486fba4e78b-kube-api-access-n9sm2\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.084299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.083955 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.084596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.084556 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.087000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.086955 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.143471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.143438 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9sm2\" (UniqueName: \"kubernetes.io/projected/afc759d7-bd00-4e59-b3cf-3486fba4e78b-kube-api-access-n9sm2\") pod \"maas-api-revoke-sub-f58f9b4f-tjgrp\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.169404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.169371 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:01.564028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:01.564002 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp"] Nov 02 21:34:01.565916 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:01.565884 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc759d7_bd00_4e59_b3cf_3486fba4e78b.slice/crio-b0c32f8f2c7f1cfe983a457425c3a4b7637a3a290464875258a5ab7abea449c8 WatchSource:0}: Error finding container b0c32f8f2c7f1cfe983a457425c3a4b7637a3a290464875258a5ab7abea449c8: Status 404 returned error can't find the container with id b0c32f8f2c7f1cfe983a457425c3a4b7637a3a290464875258a5ab7abea449c8 Nov 02 21:34:02.087777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.087733 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" event={"ID":"afc759d7-bd00-4e59-b3cf-3486fba4e78b","Type":"ContainerStarted","Data":"1f31ee32ba2a6fc11bfbac48c1ea21623c1bada006393e1d579c3e97b6aa0987"} Nov 02 21:34:02.088263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.087803 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" event={"ID":"afc759d7-bd00-4e59-b3cf-3486fba4e78b","Type":"ContainerStarted","Data":"b0c32f8f2c7f1cfe983a457425c3a4b7637a3a290464875258a5ab7abea449c8"} Nov 02 21:34:02.197938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.197841 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" podStartSLOduration=2.19782732 podStartE2EDuration="2.19782732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:02.195500075 +0000 UTC m=+1271.751128414" watchObservedRunningTime="2026-11-02 21:34:02.19782732 +0000 UTC m=+1271.753455658" Nov 02 21:34:02.491761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.491725 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk"] Nov 02 21:34:02.496843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.496819 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.499253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.499227 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc"] Nov 02 21:34:02.504041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.504022 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.535756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.535733 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b"] Nov 02 21:34:02.539812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.539792 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.588687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.588655 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk"] Nov 02 21:34:02.591863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.591837 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc"] Nov 02 21:34:02.608238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608204 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.608421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608255 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkddb\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-kube-api-access-mkddb\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.608421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608286 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608305 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.608421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608329 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792b7c65-d723-4944-9465-6691ce7a0876-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608356 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.608421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608382 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608431 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792b7c65-d723-4944-9465-6691ce7a0876-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608446 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cp4\" (UniqueName: \"kubernetes.io/projected/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-kube-api-access-47cp4\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608474 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608505 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608520 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p85s\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-kube-api-access-4p85s\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608536 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608566 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608600 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608624 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608663 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608692 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.608747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608722 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.609395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608802 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.609395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.608826 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.642020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.641982 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b"] Nov 02 21:34:02.710045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.709896 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710014 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.710214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710122 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710237 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.710328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710284 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710329 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710373 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkddb\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-kube-api-access-mkddb\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710429 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.710532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710460 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710444 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.710532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710507 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792b7c65-d723-4944-9465-6691ce7a0876-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.710732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710549 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710625 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.710832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710741 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792b7c65-d723-4944-9465-6691ce7a0876-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.710832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710764 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cp4\" (UniqueName: \"kubernetes.io/projected/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-kube-api-access-47cp4\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.710832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710816 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.711348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.710839 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.711348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.711031 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.711348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.711154 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p85s\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-kube-api-access-4p85s\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.711348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.711266 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.711348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.711330 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.711648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.711400 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.711648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.711439 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.711800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.711776 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.712405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.712046 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.713078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.713050 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.713474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.713453 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.713600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.713542 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792b7c65-d723-4944-9465-6691ce7a0876-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.714540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.714513 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792b7c65-d723-4944-9465-6691ce7a0876-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.714804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.714780 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.715205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.715183 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.715295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.715260 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.715459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.715434 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.719280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.719236 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.719280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.719249 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.719579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.719528 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.721127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.721105 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.765594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.765568 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.766821 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.766789 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.767203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.767179 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkddb\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-kube-api-access-mkddb\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:02.767266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.767233 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cp4\" (UniqueName: \"kubernetes.io/projected/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-kube-api-access-47cp4\") pod \"maas-api-revoke-sub-66013425-874jk\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.769762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.768277 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p85s\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-kube-api-access-4p85s\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.828410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.828373 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:02.837893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.837860 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:02.853071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:02.853045 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:03.104190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.103001 4441 generic.go:356] "Generic (PLEG): container finished" podID="afc759d7-bd00-4e59-b3cf-3486fba4e78b" containerID="1f31ee32ba2a6fc11bfbac48c1ea21623c1bada006393e1d579c3e97b6aa0987" exitCode=0 Nov 02 21:34:03.104190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.103307 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" event={"ID":"afc759d7-bd00-4e59-b3cf-3486fba4e78b","Type":"ContainerDied","Data":"1f31ee32ba2a6fc11bfbac48c1ea21623c1bada006393e1d579c3e97b6aa0987"} Nov 02 21:34:03.222317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.222278 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk"] Nov 02 21:34:03.227387 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:03.227356 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8417dfbf_35cf_40f9_b6e0_3bbb1454297e.slice/crio-7dad9216020e54761c69265c6ca06f9d87787d42092effc0397730d77147551e WatchSource:0}: Error finding container 7dad9216020e54761c69265c6ca06f9d87787d42092effc0397730d77147551e: Status 404 returned error can't find the container with id 7dad9216020e54761c69265c6ca06f9d87787d42092effc0397730d77147551e Nov 02 21:34:03.279710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.279677 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc"] Nov 02 21:34:03.286244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.286214 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b"] Nov 02 21:34:03.287298 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:03.287267 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792b7c65_d723_4944_9465_6691ce7a0876.slice/crio-dc6b44b49c8dfa7c7cf6c30e7ad6726b0bff0fbfc94f60fa5aecce2d70800bd3 WatchSource:0}: Error finding container dc6b44b49c8dfa7c7cf6c30e7ad6726b0bff0fbfc94f60fa5aecce2d70800bd3: Status 404 returned error can't find the container with id dc6b44b49c8dfa7c7cf6c30e7ad6726b0bff0fbfc94f60fa5aecce2d70800bd3 Nov 02 21:34:03.288439 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:03.288401 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed97436_88ee_4c2f_9418_91ec4b3e28e9.slice/crio-b2c05f85c8cf88aafd20865b5b050607b95c207f48ecfae38a38b0453c37d42e WatchSource:0}: Error finding container b2c05f85c8cf88aafd20865b5b050607b95c207f48ecfae38a38b0453c37d42e: Status 404 returned error can't find the container with id b2c05f85c8cf88aafd20865b5b050607b95c207f48ecfae38a38b0453c37d42e Nov 02 21:34:03.290424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.290390 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:03.290515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.290468 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:03.290578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.290525 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:03.290794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.290769 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:03.290856 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.290824 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:03.290856 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.290852 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:03.720741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:03.720697 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:04.150995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.148707 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" event={"ID":"1ed97436-88ee-4c2f-9418-91ec4b3e28e9","Type":"ContainerStarted","Data":"8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5"} Nov 02 21:34:04.150995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.148777 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" event={"ID":"1ed97436-88ee-4c2f-9418-91ec4b3e28e9","Type":"ContainerStarted","Data":"b2c05f85c8cf88aafd20865b5b050607b95c207f48ecfae38a38b0453c37d42e"} Nov 02 21:34:04.152200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.152170 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" event={"ID":"792b7c65-d723-4944-9465-6691ce7a0876","Type":"ContainerStarted","Data":"e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00"} Nov 02 21:34:04.152276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.152228 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" event={"ID":"792b7c65-d723-4944-9465-6691ce7a0876","Type":"ContainerStarted","Data":"dc6b44b49c8dfa7c7cf6c30e7ad6726b0bff0fbfc94f60fa5aecce2d70800bd3"} Nov 02 21:34:04.154752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.154734 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" event={"ID":"8417dfbf-35cf-40f9-b6e0-3bbb1454297e","Type":"ContainerStarted","Data":"687c30bf864989560447d87ca97558b237c2f014fb8e7c66b3c6490a7f449027"} Nov 02 21:34:04.154812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.154767 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" event={"ID":"8417dfbf-35cf-40f9-b6e0-3bbb1454297e","Type":"ContainerStarted","Data":"7dad9216020e54761c69265c6ca06f9d87787d42092effc0397730d77147551e"} Nov 02 21:34:04.315997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.315738 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podStartSLOduration=2.315718663 podStartE2EDuration="2.315718663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:04.264575796 +0000 UTC m=+1273.820204133" watchObservedRunningTime="2026-11-02 21:34:04.315718663 +0000 UTC m=+1273.871347000" Nov 02 21:34:04.555662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.555607 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" podStartSLOduration=2.555587545 podStartE2EDuration="2.555587545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:04.55523694 +0000 UTC m=+1274.110865282" watchObservedRunningTime="2026-11-02 21:34:04.555587545 +0000 UTC m=+1274.111215883" Nov 02 21:34:04.557377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.557332 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podStartSLOduration=2.557320782 podStartE2EDuration="2.557320782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:04.391947892 +0000 UTC m=+1273.947576242" watchObservedRunningTime="2026-11-02 21:34:04.557320782 +0000 UTC m=+1274.112949119" Nov 02 21:34:04.640207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.640170 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:04.757849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.756863 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc759d7-bd00-4e59-b3cf-3486fba4e78b-kube-api-access-n9sm2\" (UniqueName: \"kubernetes.io/projected/afc759d7-bd00-4e59-b3cf-3486fba4e78b-kube-api-access-n9sm2\") pod \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " Nov 02 21:34:04.757849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.756932 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-openshift-ca\") pod \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " Nov 02 21:34:04.757849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.757027 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-xks-ca\") pod \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\" (UID: \"afc759d7-bd00-4e59-b3cf-3486fba4e78b\") " Nov 02 21:34:04.757849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.757565 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-openshift-ca" pod "afc759d7-bd00-4e59-b3cf-3486fba4e78b" (UID: "afc759d7-bd00-4e59-b3cf-3486fba4e78b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:04.757849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.757785 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:04.761000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.759944 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc759d7-bd00-4e59-b3cf-3486fba4e78b-kube-api-access-n9sm2" pod "afc759d7-bd00-4e59-b3cf-3486fba4e78b" (UID: "afc759d7-bd00-4e59-b3cf-3486fba4e78b"). InnerVolumeSpecName "kube-api-access-n9sm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:04.761000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.760049 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-xks-ca" pod "afc759d7-bd00-4e59-b3cf-3486fba4e78b" (UID: "afc759d7-bd00-4e59-b3cf-3486fba4e78b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:04.839329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.839260 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:04.840777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.840408 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:34:04.840777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.840450 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 21:34:04.853662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.853642 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:04.854568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.854478 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:34:04.854568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.854514 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 21:34:04.859626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.859604 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9sm2\" (UniqueName: \"kubernetes.io/projected/afc759d7-bd00-4e59-b3cf-3486fba4e78b-kube-api-access-n9sm2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:04.859692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:04.859628 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc759d7-bd00-4e59-b3cf-3486fba4e78b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:05.165275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.165187 4441 generic.go:356] "Generic (PLEG): container finished" podID="8417dfbf-35cf-40f9-b6e0-3bbb1454297e" containerID="687c30bf864989560447d87ca97558b237c2f014fb8e7c66b3c6490a7f449027" exitCode=0 Nov 02 21:34:05.165722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.165282 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" event={"ID":"8417dfbf-35cf-40f9-b6e0-3bbb1454297e","Type":"ContainerDied","Data":"687c30bf864989560447d87ca97558b237c2f014fb8e7c66b3c6490a7f449027"} Nov 02 21:34:05.167329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.167305 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" Nov 02 21:34:05.167461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.167443 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp" event={"ID":"afc759d7-bd00-4e59-b3cf-3486fba4e78b","Type":"ContainerDied","Data":"b0c32f8f2c7f1cfe983a457425c3a4b7637a3a290464875258a5ab7abea449c8"} Nov 02 21:34:05.167546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.167532 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0c32f8f2c7f1cfe983a457425c3a4b7637a3a290464875258a5ab7abea449c8" Nov 02 21:34:05.842031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.841077 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:34:05.842031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.841132 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 21:34:05.853419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.853364 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:34:05.853485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:05.853415 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 21:34:06.137701 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.137601 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp"] Nov 02 21:34:06.138256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.138236 4441 cpu_manager.go:414] "containerMap: removing container" podUID="afc759d7-bd00-4e59-b3cf-3486fba4e78b" containerName="revoke-keys" Nov 02 21:34:06.138256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.138254 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc759d7-bd00-4e59-b3cf-3486fba4e78b" containerName="revoke-keys" Nov 02 21:34:06.138454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.138382 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc759d7-bd00-4e59-b3cf-3486fba4e78b" containerName="revoke-keys" Nov 02 21:34:06.142771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.142747 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.245549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.244882 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.245549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.245124 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cggn5\" (UniqueName: \"kubernetes.io/projected/75f25a0d-bad9-42c3-bc04-f92933273134-kube-api-access-cggn5\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.245549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.245192 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.252223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.252198 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp"] Nov 02 21:34:06.347934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.347890 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cggn5\" (UniqueName: \"kubernetes.io/projected/75f25a0d-bad9-42c3-bc04-f92933273134-kube-api-access-cggn5\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.348179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.348060 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.348401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.348374 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.349332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.349308 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.350774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.350745 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.455535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.455356 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cggn5\" (UniqueName: \"kubernetes.io/projected/75f25a0d-bad9-42c3-bc04-f92933273134-kube-api-access-cggn5\") pod \"maas-api-revoke-sub-e670b198-v52rp\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.455535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.455512 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:06.645731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.645160 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:06.655720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.655693 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-openshift-ca\") pod \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " Nov 02 21:34:06.655852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.655737 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-kube-api-access-47cp4\" (UniqueName: \"kubernetes.io/projected/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-kube-api-access-47cp4\") pod \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " Nov 02 21:34:06.655852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.655795 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-xks-ca\") pod \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\" (UID: \"8417dfbf-35cf-40f9-b6e0-3bbb1454297e\") " Nov 02 21:34:06.656880 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.656734 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-openshift-ca" pod "8417dfbf-35cf-40f9-b6e0-3bbb1454297e" (UID: "8417dfbf-35cf-40f9-b6e0-3bbb1454297e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:06.658548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.658521 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-kube-api-access-47cp4" pod "8417dfbf-35cf-40f9-b6e0-3bbb1454297e" (UID: "8417dfbf-35cf-40f9-b6e0-3bbb1454297e"). InnerVolumeSpecName "kube-api-access-47cp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:06.658734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.658708 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-xks-ca" pod "8417dfbf-35cf-40f9-b6e0-3bbb1454297e" (UID: "8417dfbf-35cf-40f9-b6e0-3bbb1454297e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:06.759082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.759057 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:06.759201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.759191 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cp4\" (UniqueName: \"kubernetes.io/projected/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-kube-api-access-47cp4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:06.759281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.759272 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8417dfbf-35cf-40f9-b6e0-3bbb1454297e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:06.838655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.838618 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:34:06.838907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.838877 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 21:34:06.855445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.855417 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:34:06.855571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:06.855464 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 21:34:07.080990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.079769 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp"] Nov 02 21:34:07.257114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.257078 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" event={"ID":"75f25a0d-bad9-42c3-bc04-f92933273134","Type":"ContainerStarted","Data":"31247d62098cfcf60ab42970c138a0ec57e6c644fd0571c4be7458891e2f3ebd"} Nov 02 21:34:07.258635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.258612 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" Nov 02 21:34:07.258737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.258652 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk" event={"ID":"8417dfbf-35cf-40f9-b6e0-3bbb1454297e","Type":"ContainerDied","Data":"7dad9216020e54761c69265c6ca06f9d87787d42092effc0397730d77147551e"} Nov 02 21:34:07.258737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.258696 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dad9216020e54761c69265c6ca06f9d87787d42092effc0397730d77147551e" Nov 02 21:34:07.838494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.838463 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:34:07.838680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.838660 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 21:34:07.853463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.853438 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:34:07.853649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:07.853629 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 21:34:08.268369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:08.268335 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" event={"ID":"75f25a0d-bad9-42c3-bc04-f92933273134","Type":"ContainerStarted","Data":"e7527f830864d6997f601cf7c237446133ce0ad1cbdd710885790eb921dd885d"} Nov 02 21:34:08.429015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:08.428262 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:08.447388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:08.447249 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" podStartSLOduration=2.447226056 podStartE2EDuration="2.447226056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:08.394684355 +0000 UTC m=+1277.950312730" watchObservedRunningTime="2026-11-02 21:34:08.447226056 +0000 UTC m=+1278.002854394" Nov 02 21:34:08.838994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:08.838423 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:34:08.838994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:08.838482 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 21:34:08.853834 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:08.853788 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:34:08.853956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:08.853853 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 21:34:09.045593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.045531 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4"] Nov 02 21:34:09.050662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.050637 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fde3573-dbwr4"] Nov 02 21:34:09.334534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.334492 4441 generic.go:356] "Generic (PLEG): container finished" podID="75f25a0d-bad9-42c3-bc04-f92933273134" containerID="e7527f830864d6997f601cf7c237446133ce0ad1cbdd710885790eb921dd885d" exitCode=0 Nov 02 21:34:09.335096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.334867 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" event={"ID":"75f25a0d-bad9-42c3-bc04-f92933273134","Type":"ContainerDied","Data":"e7527f830864d6997f601cf7c237446133ce0ad1cbdd710885790eb921dd885d"} Nov 02 21:34:09.838582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.838545 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:34:09.838770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.838598 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 21:34:09.853665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.853633 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:34:09.853783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:09.853691 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 21:34:10.545376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.545350 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:10.578242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.578187 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-openshift-ca\") pod \"75f25a0d-bad9-42c3-bc04-f92933273134\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " Nov 02 21:34:10.578668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.578410 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-xks-ca\") pod \"75f25a0d-bad9-42c3-bc04-f92933273134\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " Nov 02 21:34:10.578668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.578542 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f25a0d-bad9-42c3-bc04-f92933273134-kube-api-access-cggn5\" (UniqueName: \"kubernetes.io/projected/75f25a0d-bad9-42c3-bc04-f92933273134-kube-api-access-cggn5\") pod \"75f25a0d-bad9-42c3-bc04-f92933273134\" (UID: \"75f25a0d-bad9-42c3-bc04-f92933273134\") " Nov 02 21:34:10.579380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.579333 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-openshift-ca" pod "75f25a0d-bad9-42c3-bc04-f92933273134" (UID: "75f25a0d-bad9-42c3-bc04-f92933273134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:10.581594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.581547 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-xks-ca" pod "75f25a0d-bad9-42c3-bc04-f92933273134" (UID: "75f25a0d-bad9-42c3-bc04-f92933273134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:10.581845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.581820 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f25a0d-bad9-42c3-bc04-f92933273134-kube-api-access-cggn5" pod "75f25a0d-bad9-42c3-bc04-f92933273134" (UID: "75f25a0d-bad9-42c3-bc04-f92933273134"). InnerVolumeSpecName "kube-api-access-cggn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:10.592493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.592460 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9017c47f-0410-44ca-a546-f4be35713947" path="/var/lib/kubelet/pods/9017c47f-0410-44ca-a546-f4be35713947/volumes" Nov 02 21:34:10.636007 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:34:10.635807 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:34:10.681645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.681157 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:10.681645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.681203 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cggn5\" (UniqueName: \"kubernetes.io/projected/75f25a0d-bad9-42c3-bc04-f92933273134-kube-api-access-cggn5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:10.681645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:10.681221 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f25a0d-bad9-42c3-bc04-f92933273134-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:11.068030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.067997 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:11.068797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.068771 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:11.069276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.069259 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:11.069767 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.069705 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:11.069896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.069878 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:34:11.070578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.070557 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:34:11.357691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.357610 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" Nov 02 21:34:11.357691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.357624 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp" event={"ID":"75f25a0d-bad9-42c3-bc04-f92933273134","Type":"ContainerDied","Data":"31247d62098cfcf60ab42970c138a0ec57e6c644fd0571c4be7458891e2f3ebd"} Nov 02 21:34:11.357885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:11.357700 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31247d62098cfcf60ab42970c138a0ec57e6c644fd0571c4be7458891e2f3ebd" Nov 02 21:34:12.639105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.639077 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42"] Nov 02 21:34:12.639674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.639659 4441 cpu_manager.go:414] "containerMap: removing container" podUID="75f25a0d-bad9-42c3-bc04-f92933273134" containerName="revoke-keys" Nov 02 21:34:12.639739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.639676 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f25a0d-bad9-42c3-bc04-f92933273134" containerName="revoke-keys" Nov 02 21:34:12.639739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.639706 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8417dfbf-35cf-40f9-b6e0-3bbb1454297e" containerName="revoke-keys" Nov 02 21:34:12.639739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.639711 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8417dfbf-35cf-40f9-b6e0-3bbb1454297e" containerName="revoke-keys" Nov 02 21:34:12.639837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.639789 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8417dfbf-35cf-40f9-b6e0-3bbb1454297e" containerName="revoke-keys" Nov 02 21:34:12.639837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.639800 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f25a0d-bad9-42c3-bc04-f92933273134" containerName="revoke-keys" Nov 02 21:34:12.643607 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.643588 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.693504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.693471 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42"] Nov 02 21:34:12.705157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.705125 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.705298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.705170 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.705340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.705309 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktpd8\" (UniqueName: \"kubernetes.io/projected/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-kube-api-access-ktpd8\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.733749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.733714 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:34:12.733749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.733726 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:34:12.733986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.733919 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:34:12.740048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.740019 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:34:12.806405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.806376 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktpd8\" (UniqueName: \"kubernetes.io/projected/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-kube-api-access-ktpd8\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.806554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.806522 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.806624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.806554 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.807253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.807232 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.808822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.808803 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.860436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.860412 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktpd8\" (UniqueName: \"kubernetes.io/projected/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-kube-api-access-ktpd8\") pod \"maas-api-revoke-sub-a5218240-scx42\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.961324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.961261 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:12.994610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:12.994586 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:13.065093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:13.065048 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5"] Nov 02 21:34:13.077070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:13.077039 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533db7e7-48kp5"] Nov 02 21:34:13.344875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:13.344845 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42"] Nov 02 21:34:13.348679 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:13.348648 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4387fa0b_7c13_4ff2_b852_bc7bca78bac9.slice/crio-854f8209a87bc2c22a5edc338e7de4841382d3f9c9c782376863de48082df3fd WatchSource:0}: Error finding container 854f8209a87bc2c22a5edc338e7de4841382d3f9c9c782376863de48082df3fd: Status 404 returned error can't find the container with id 854f8209a87bc2c22a5edc338e7de4841382d3f9c9c782376863de48082df3fd Nov 02 21:34:13.375175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:13.375150 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" event={"ID":"4387fa0b-7c13-4ff2-b852-bc7bca78bac9","Type":"ContainerStarted","Data":"854f8209a87bc2c22a5edc338e7de4841382d3f9c9c782376863de48082df3fd"} Nov 02 21:34:14.386628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:14.386597 4441 generic.go:356] "Generic (PLEG): container finished" podID="4387fa0b-7c13-4ff2-b852-bc7bca78bac9" containerID="99c13d8a00e0319c3717e808e42c8a7ea83d037777ebb09ce8af7ecab9ac5cb2" exitCode=0 Nov 02 21:34:14.387066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:14.386690 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" event={"ID":"4387fa0b-7c13-4ff2-b852-bc7bca78bac9","Type":"ContainerDied","Data":"99c13d8a00e0319c3717e808e42c8a7ea83d037777ebb09ce8af7ecab9ac5cb2"} Nov 02 21:34:14.584314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:14.584282 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240c5a07-0957-4f3e-94df-d35d7dfc420b" path="/var/lib/kubelet/pods/240c5a07-0957-4f3e-94df-d35d7dfc420b/volumes" Nov 02 21:34:15.564144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.564121 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:15.641951 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.641921 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-openshift-ca\") pod \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " Nov 02 21:34:15.642130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.641993 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-kube-api-access-ktpd8\" (UniqueName: \"kubernetes.io/projected/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-kube-api-access-ktpd8\") pod \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " Nov 02 21:34:15.642130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.642030 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-xks-ca\") pod \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\" (UID: \"4387fa0b-7c13-4ff2-b852-bc7bca78bac9\") " Nov 02 21:34:15.642299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.642271 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-openshift-ca" pod "4387fa0b-7c13-4ff2-b852-bc7bca78bac9" (UID: "4387fa0b-7c13-4ff2-b852-bc7bca78bac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:15.642766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.642736 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:15.644258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.644237 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-xks-ca" pod "4387fa0b-7c13-4ff2-b852-bc7bca78bac9" (UID: "4387fa0b-7c13-4ff2-b852-bc7bca78bac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:15.644574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.644557 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-kube-api-access-ktpd8" pod "4387fa0b-7c13-4ff2-b852-bc7bca78bac9" (UID: "4387fa0b-7c13-4ff2-b852-bc7bca78bac9"). InnerVolumeSpecName "kube-api-access-ktpd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:15.744021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.743987 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktpd8\" (UniqueName: \"kubernetes.io/projected/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-kube-api-access-ktpd8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:15.744021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:15.744023 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4387fa0b-7c13-4ff2-b852-bc7bca78bac9-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:16.407633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:16.407568 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" event={"ID":"4387fa0b-7c13-4ff2-b852-bc7bca78bac9","Type":"ContainerDied","Data":"854f8209a87bc2c22a5edc338e7de4841382d3f9c9c782376863de48082df3fd"} Nov 02 21:34:16.407633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:16.407623 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="854f8209a87bc2c22a5edc338e7de4841382d3f9c9c782376863de48082df3fd" Nov 02 21:34:16.407633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:16.407577 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42" Nov 02 21:34:17.059326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.059296 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld"] Nov 02 21:34:17.059925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.059909 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4387fa0b-7c13-4ff2-b852-bc7bca78bac9" containerName="revoke-keys" Nov 02 21:34:17.060005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.059929 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4387fa0b-7c13-4ff2-b852-bc7bca78bac9" containerName="revoke-keys" Nov 02 21:34:17.060125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.060113 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4387fa0b-7c13-4ff2-b852-bc7bca78bac9" containerName="revoke-keys" Nov 02 21:34:17.064153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.064125 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.161747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.161712 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mltn\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-kube-api-access-4mltn\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.161898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.161776 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.161898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.161887 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.162025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.161941 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.162092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.162074 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.162143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.162117 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.162196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.162170 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.162308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.162289 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.162348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.162323 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.201731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.201686 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld"] Nov 02 21:34:17.263246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263202 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263261 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263285 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263316 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263391 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263413 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263559 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mltn\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-kube-api-access-4mltn\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263627 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263718 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263757 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.263873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263817 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.264104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.263873 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.264104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.264053 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.264188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.264166 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.265917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.265890 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.266141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.266124 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.318428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.318354 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mltn\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-kube-api-access-4mltn\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.319076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.319054 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.378464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.378433 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:17.510963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.510937 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:17.831721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.831687 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld"] Nov 02 21:34:17.836840 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:17.836809 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caabab8_b0ee_4748_a24f_6bdeb1149f93.slice/crio-876c0338ac7722a559d45a329b8c08660eefc0b1d1532dba4608e6de3a88e900 WatchSource:0}: Error finding container 876c0338ac7722a559d45a329b8c08660eefc0b1d1532dba4608e6de3a88e900: Status 404 returned error can't find the container with id 876c0338ac7722a559d45a329b8c08660eefc0b1d1532dba4608e6de3a88e900 Nov 02 21:34:17.839473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.839439 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:17.839563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.839518 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:17.839641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.839565 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:17.898660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.898632 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd"] Nov 02 21:34:17.903183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.903164 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.972422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.972392 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd"] Nov 02 21:34:17.974797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.974761 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.974949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.974824 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkfr\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-kube-api-access-9qkfr\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.974949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.974922 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.975090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.975019 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bc6189-eae4-4b30-9822-2d890886aa31-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.975090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.975080 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.975210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.975117 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.975262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.975233 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.975312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.975271 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:17.975312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:17.975298 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.076996 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077066 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077095 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077213 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077257 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkfr\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-kube-api-access-9qkfr\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077512 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077602 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bc6189-eae4-4b30-9822-2d890886aa31-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077671 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.077777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077715 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.078007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077874 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.078007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.077600 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.078204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.078178 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.078302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.078207 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.078302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.078259 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bc6189-eae4-4b30-9822-2d890886aa31-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.079623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.079599 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.080192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.080168 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.131443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.131370 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.131443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.131419 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkfr\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-kube-api-access-9qkfr\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.219607 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.219571 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:18.246905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.246831 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6"] Nov 02 21:34:18.253327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.253302 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.332526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.331755 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6"] Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.381750 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.381799 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.381883 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73714c5-4059-4c1f-924c-6883b6852311-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.381913 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.381952 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-kube-api-access-6d5kv\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.382140 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.382216 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73714c5-4059-4c1f-924c-6883b6852311-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.382253 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.384493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.382352 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.445998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.445434 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" event={"ID":"4caabab8-b0ee-4748-a24f-6bdeb1149f93","Type":"ContainerStarted","Data":"1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442"} Nov 02 21:34:18.445998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.445509 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" event={"ID":"4caabab8-b0ee-4748-a24f-6bdeb1149f93","Type":"ContainerStarted","Data":"876c0338ac7722a559d45a329b8c08660eefc0b1d1532dba4608e6de3a88e900"} Nov 02 21:34:18.492034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.485822 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73714c5-4059-4c1f-924c-6883b6852311-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.492322 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.499086 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-kube-api-access-6d5kv\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.499163 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.499245 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73714c5-4059-4c1f-924c-6883b6852311-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.499283 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.499402 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.499716 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.499857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.499842 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.500065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.498001 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.500137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.498865 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73714c5-4059-4c1f-924c-6883b6852311-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.500404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.500372 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.500600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.500572 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.500682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.500633 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.501447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.501175 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73714c5-4059-4c1f-924c-6883b6852311-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.504456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.504430 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.553583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.553543 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-kube-api-access-6d5kv\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.553583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.553561 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.565212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.565129 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podStartSLOduration=1.5651165790000001 podStartE2EDuration="1.565116579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:18.560858236 +0000 UTC m=+1288.116486573" watchObservedRunningTime="2026-11-02 21:34:18.565116579 +0000 UTC m=+1288.120744915" Nov 02 21:34:18.568094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.568072 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:18.642869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.642793 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd"] Nov 02 21:34:18.647960 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:18.647918 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bc6189_eae4_4b30_9822_2d890886aa31.slice/crio-85a1033febb41c3b4ac2f390211b8a323e8623c95184ed8a6e56040263ef184c WatchSource:0}: Error finding container 85a1033febb41c3b4ac2f390211b8a323e8623c95184ed8a6e56040263ef184c: Status 404 returned error can't find the container with id 85a1033febb41c3b4ac2f390211b8a323e8623c95184ed8a6e56040263ef184c Nov 02 21:34:18.650723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.650690 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:18.650836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.650759 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:18.650836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.650803 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:18.998300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:18.998209 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6"] Nov 02 21:34:19.005146 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:19.005113 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73714c5_4059_4c1f_924c_6883b6852311.slice/crio-8eb4a79f102ca1477c76e31c8eccc0bc4cc7b2388ed7d252865053afa80016e4 WatchSource:0}: Error finding container 8eb4a79f102ca1477c76e31c8eccc0bc4cc7b2388ed7d252865053afa80016e4: Status 404 returned error can't find the container with id 8eb4a79f102ca1477c76e31c8eccc0bc4cc7b2388ed7d252865053afa80016e4 Nov 02 21:34:19.008537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.008501 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:19.008613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.008569 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:19.008613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.008597 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:19.384992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.384286 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:19.393991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.392074 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:34:19.393991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.392134 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 21:34:19.461405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.461370 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" event={"ID":"d73714c5-4059-4c1f-924c-6883b6852311","Type":"ContainerStarted","Data":"38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8"} Nov 02 21:34:19.461562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.461434 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" event={"ID":"d73714c5-4059-4c1f-924c-6883b6852311","Type":"ContainerStarted","Data":"8eb4a79f102ca1477c76e31c8eccc0bc4cc7b2388ed7d252865053afa80016e4"} Nov 02 21:34:19.463997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.463942 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" event={"ID":"e7bc6189-eae4-4b30-9822-2d890886aa31","Type":"ContainerStarted","Data":"a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69"} Nov 02 21:34:19.464128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.463998 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" event={"ID":"e7bc6189-eae4-4b30-9822-2d890886aa31","Type":"ContainerStarted","Data":"85a1033febb41c3b4ac2f390211b8a323e8623c95184ed8a6e56040263ef184c"} Nov 02 21:34:19.569039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.569009 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:19.747608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.747475 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podStartSLOduration=1.74744893 podStartE2EDuration="1.74744893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:19.607272102 +0000 UTC m=+1289.162900452" watchObservedRunningTime="2026-11-02 21:34:19.74744893 +0000 UTC m=+1289.303077267" Nov 02 21:34:19.754356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:19.754300 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podStartSLOduration=2.754285287 podStartE2EDuration="2.754285287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:19.752361821 +0000 UTC m=+1289.307990169" watchObservedRunningTime="2026-11-02 21:34:19.754285287 +0000 UTC m=+1289.309913617" Nov 02 21:34:20.223991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:20.223173 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:20.223991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:20.223390 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:34:20.223991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:20.223438 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 21:34:20.390990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:20.387493 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:34:20.390990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:20.387586 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 21:34:20.575997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:20.575866 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:34:20.575997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:20.575939 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 21:34:21.222992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:21.222318 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:34:21.222992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:21.222380 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 21:34:21.382991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:21.382293 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:34:21.382991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:21.382355 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 21:34:21.590034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:21.568916 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:34:21.590034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:21.568996 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 21:34:22.287023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:22.260322 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:34:22.287023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:22.260381 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 21:34:22.408993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:22.408458 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:34:22.408993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:22.408528 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 21:34:22.636993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:22.635087 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:34:22.636993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:22.635152 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 21:34:23.058077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.056879 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:23.230001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.221998 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:34:23.230001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.222065 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 21:34:23.363182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.363089 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.403026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.385739 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:34:23.403026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.385795 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 21:34:23.506021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.498507 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:34:23.542112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.542069 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.568221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.567516 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.608724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.608687 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:34:23.609476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.609445 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 21:34:23.638175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:23.638045 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:24.167960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.167912 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk"] Nov 02 21:34:24.235372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.235320 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:34:24.235615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.235589 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 21:34:24.255439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.255407 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.259286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.257136 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk"] Nov 02 21:34:24.265895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.265862 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a467613-ece7-441a-a190-73a008fd332b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.266178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.266153 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a467613-ece7-441a-a190-73a008fd332b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.266322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.266304 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqbk\" (UniqueName: \"kubernetes.io/projected/3a467613-ece7-441a-a190-73a008fd332b-kube-api-access-8tqbk\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.367288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.367255 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a467613-ece7-441a-a190-73a008fd332b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.367464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.367352 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a467613-ece7-441a-a190-73a008fd332b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.367464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.367372 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqbk\" (UniqueName: \"kubernetes.io/projected/3a467613-ece7-441a-a190-73a008fd332b-kube-api-access-8tqbk\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.385482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.385449 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:34:24.385703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.385676 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 21:34:24.386272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.386253 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:34:24.397073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.397042 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:34:24.429518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.429431 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:34:24.429805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.429785 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:34:24.450027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.438489 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a467613-ece7-441a-a190-73a008fd332b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.450027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.438733 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a467613-ece7-441a-a190-73a008fd332b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.458625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.457404 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqbk\" (UniqueName: \"kubernetes.io/projected/3a467613-ece7-441a-a190-73a008fd332b-kube-api-access-8tqbk\") pod \"maas-api-revoke-sub-01dce43d-bwnnk\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.568926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.568891 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:34:24.569149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.568940 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 21:34:24.641805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.641772 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:24.643527 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.643490 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf"] Nov 02 21:34:24.649238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.649222 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.687038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.682808 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2q78\" (UniqueName: \"kubernetes.io/projected/0db76fec-1d0f-448a-aecd-b0f00648cef8-kube-api-access-q2q78\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.687392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.687364 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.723992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.687642 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.724701 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.708701 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf"] Nov 02 21:34:24.859355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.859318 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.859668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.859642 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2q78\" (UniqueName: \"kubernetes.io/projected/0db76fec-1d0f-448a-aecd-b0f00648cef8-kube-api-access-q2q78\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.859938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.859916 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.862832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.860932 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.864526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.864505 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:24.964836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:24.964767 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2q78\" (UniqueName: \"kubernetes.io/projected/0db76fec-1d0f-448a-aecd-b0f00648cef8-kube-api-access-q2q78\") pod \"maas-api-revoke-sub-848cbd60-g5sdf\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:25.063010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.059661 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:25.227006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.226073 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:34:25.227006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.226134 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 21:34:25.398259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.397301 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:34:25.398259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.397359 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 21:34:25.548004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.540957 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk"] Nov 02 21:34:25.568413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.568341 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:34:25.568413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.568380 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 21:34:25.864058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:25.864027 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf"] Nov 02 21:34:25.864608 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:25.864571 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db76fec_1d0f_448a_aecd_b0f00648cef8.slice/crio-bf5ccf455ba40f37623f1f7c4426d0d18e313544b3f69b7f53f2f2144a0be608 WatchSource:0}: Error finding container bf5ccf455ba40f37623f1f7c4426d0d18e313544b3f69b7f53f2f2144a0be608: Status 404 returned error can't find the container with id bf5ccf455ba40f37623f1f7c4426d0d18e313544b3f69b7f53f2f2144a0be608 Nov 02 21:34:26.155282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.155172 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" event={"ID":"3a467613-ece7-441a-a190-73a008fd332b","Type":"ContainerStarted","Data":"eea66cd686759cc4ad61d0a81943e07bc98402330debcaa2131487e4831252ec"} Nov 02 21:34:26.155282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.155248 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" event={"ID":"3a467613-ece7-441a-a190-73a008fd332b","Type":"ContainerStarted","Data":"7b4b7a212b70bd09b56a5cb9a208ce0b5aed97e08ed977a609b5f0f7b26a4f8f"} Nov 02 21:34:26.157267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.157231 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" event={"ID":"0db76fec-1d0f-448a-aecd-b0f00648cef8","Type":"ContainerStarted","Data":"bf5ccf455ba40f37623f1f7c4426d0d18e313544b3f69b7f53f2f2144a0be608"} Nov 02 21:34:26.220920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.220871 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:34:26.221105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.220931 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 21:34:26.265835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.265777 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" podStartSLOduration=2.265762977 podStartE2EDuration="2.265762977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:26.262394893 +0000 UTC m=+1295.818023226" watchObservedRunningTime="2026-11-02 21:34:26.265762977 +0000 UTC m=+1295.821391313" Nov 02 21:34:26.386090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.385980 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:26.386808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.386757 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:26.387858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.387829 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:34:26.583675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.582425 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:34:26.583675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:26.582486 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 21:34:27.049382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.049338 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f"] Nov 02 21:34:27.057997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.057951 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df54a85e-znk6f"] Nov 02 21:34:27.170897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.170861 4441 generic.go:356] "Generic (PLEG): container finished" podID="3a467613-ece7-441a-a190-73a008fd332b" containerID="eea66cd686759cc4ad61d0a81943e07bc98402330debcaa2131487e4831252ec" exitCode=0 Nov 02 21:34:27.171096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.170917 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" event={"ID":"3a467613-ece7-441a-a190-73a008fd332b","Type":"ContainerDied","Data":"eea66cd686759cc4ad61d0a81943e07bc98402330debcaa2131487e4831252ec"} Nov 02 21:34:27.173112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.173084 4441 generic.go:356] "Generic (PLEG): container finished" podID="0db76fec-1d0f-448a-aecd-b0f00648cef8" containerID="895e8c77cb1a975af788b8c3e75e6d59d7185fb72edab104c222f50b2d2ffa1d" exitCode=0 Nov 02 21:34:27.173231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.173175 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" event={"ID":"0db76fec-1d0f-448a-aecd-b0f00648cef8","Type":"ContainerDied","Data":"895e8c77cb1a975af788b8c3e75e6d59d7185fb72edab104c222f50b2d2ffa1d"} Nov 02 21:34:27.228035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.227985 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:27.229215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.229171 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:27.230282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.230261 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:34:27.573582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.573557 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:27.574086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.574064 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:27.575169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:27.575145 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:34:28.046702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.046626 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd"] Nov 02 21:34:28.051781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.051751 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad6d08a-tdlmd"] Nov 02 21:34:28.366885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.366864 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:28.394061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.394030 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:28.422366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.422341 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:28.486088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.485304 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db76fec-1d0f-448a-aecd-b0f00648cef8-kube-api-access-q2q78\" (UniqueName: \"kubernetes.io/projected/0db76fec-1d0f-448a-aecd-b0f00648cef8-kube-api-access-q2q78\") pod \"0db76fec-1d0f-448a-aecd-b0f00648cef8\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " Nov 02 21:34:28.486088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.485459 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a467613-ece7-441a-a190-73a008fd332b-kube-api-access-8tqbk\" (UniqueName: \"kubernetes.io/projected/3a467613-ece7-441a-a190-73a008fd332b-kube-api-access-8tqbk\") pod \"3a467613-ece7-441a-a190-73a008fd332b\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " Nov 02 21:34:28.486088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.485519 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-xks-ca\") pod \"0db76fec-1d0f-448a-aecd-b0f00648cef8\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " Nov 02 21:34:28.486088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.485642 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a467613-ece7-441a-a190-73a008fd332b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a467613-ece7-441a-a190-73a008fd332b-maas-api-openshift-ca\") pod \"3a467613-ece7-441a-a190-73a008fd332b\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " Nov 02 21:34:28.486088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.485674 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-openshift-ca\") pod \"0db76fec-1d0f-448a-aecd-b0f00648cef8\" (UID: \"0db76fec-1d0f-448a-aecd-b0f00648cef8\") " Nov 02 21:34:28.486088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.485731 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a467613-ece7-441a-a190-73a008fd332b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a467613-ece7-441a-a190-73a008fd332b-maas-api-xks-ca\") pod \"3a467613-ece7-441a-a190-73a008fd332b\" (UID: \"3a467613-ece7-441a-a190-73a008fd332b\") " Nov 02 21:34:28.490363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.490329 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-openshift-ca" pod "0db76fec-1d0f-448a-aecd-b0f00648cef8" (UID: "0db76fec-1d0f-448a-aecd-b0f00648cef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:28.490667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.490643 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a467613-ece7-441a-a190-73a008fd332b-maas-api-openshift-ca" pod "3a467613-ece7-441a-a190-73a008fd332b" (UID: "3a467613-ece7-441a-a190-73a008fd332b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:28.520541 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.520504 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a467613-ece7-441a-a190-73a008fd332b-kube-api-access-8tqbk" pod "3a467613-ece7-441a-a190-73a008fd332b" (UID: "3a467613-ece7-441a-a190-73a008fd332b"). InnerVolumeSpecName "kube-api-access-8tqbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:28.523620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.523571 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db76fec-1d0f-448a-aecd-b0f00648cef8-kube-api-access-q2q78" pod "0db76fec-1d0f-448a-aecd-b0f00648cef8" (UID: "0db76fec-1d0f-448a-aecd-b0f00648cef8"). InnerVolumeSpecName "kube-api-access-q2q78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:28.523620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.523576 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a467613-ece7-441a-a190-73a008fd332b-maas-api-xks-ca" pod "3a467613-ece7-441a-a190-73a008fd332b" (UID: "3a467613-ece7-441a-a190-73a008fd332b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:28.525791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.525767 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-xks-ca" pod "0db76fec-1d0f-448a-aecd-b0f00648cef8" (UID: "0db76fec-1d0f-448a-aecd-b0f00648cef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:28.592404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.592361 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqbk\" (UniqueName: \"kubernetes.io/projected/3a467613-ece7-441a-a190-73a008fd332b-kube-api-access-8tqbk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:28.592404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.592400 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:28.592789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.592419 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a467613-ece7-441a-a190-73a008fd332b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:28.592789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.592434 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db76fec-1d0f-448a-aecd-b0f00648cef8-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:28.592789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.592448 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a467613-ece7-441a-a190-73a008fd332b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:28.592789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.592463 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2q78\" (UniqueName: \"kubernetes.io/projected/0db76fec-1d0f-448a-aecd-b0f00648cef8-kube-api-access-q2q78\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:28.602264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.602175 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190f802f-8675-49bb-884b-65a57742f357" path="/var/lib/kubelet/pods/190f802f-8675-49bb-884b-65a57742f357/volumes" Nov 02 21:34:28.602842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:28.602815 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7727674b-a53c-4ca5-a06e-98ddeecd5148" path="/var/lib/kubelet/pods/7727674b-a53c-4ca5-a06e-98ddeecd5148/volumes" Nov 02 21:34:29.200870 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:29.200840 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" Nov 02 21:34:29.202033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:29.201055 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf" event={"ID":"0db76fec-1d0f-448a-aecd-b0f00648cef8","Type":"ContainerDied","Data":"bf5ccf455ba40f37623f1f7c4426d0d18e313544b3f69b7f53f2f2144a0be608"} Nov 02 21:34:29.202149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:29.202118 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5ccf455ba40f37623f1f7c4426d0d18e313544b3f69b7f53f2f2144a0be608" Nov 02 21:34:29.204586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:29.204196 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" Nov 02 21:34:29.204586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:29.204666 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk" event={"ID":"3a467613-ece7-441a-a190-73a008fd332b","Type":"ContainerDied","Data":"7b4b7a212b70bd09b56a5cb9a208ce0b5aed97e08ed977a609b5f0f7b26a4f8f"} Nov 02 21:34:29.204586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:29.204708 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b4b7a212b70bd09b56a5cb9a208ce0b5aed97e08ed977a609b5f0f7b26a4f8f" Nov 02 21:34:30.044916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.044879 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8"] Nov 02 21:34:30.049808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.049784 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99bab561-66hn8"] Nov 02 21:34:30.586980 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.586938 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299d18d5-42dc-4eb5-990e-c069d11a414e" path="/var/lib/kubelet/pods/299d18d5-42dc-4eb5-990e-c069d11a414e/volumes" Nov 02 21:34:30.860571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.860493 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554"] Nov 02 21:34:30.861243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.861227 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3a467613-ece7-441a-a190-73a008fd332b" containerName="revoke-keys" Nov 02 21:34:30.861298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.861244 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a467613-ece7-441a-a190-73a008fd332b" containerName="revoke-keys" Nov 02 21:34:30.861298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.861255 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0db76fec-1d0f-448a-aecd-b0f00648cef8" containerName="revoke-keys" Nov 02 21:34:30.861298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.861260 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db76fec-1d0f-448a-aecd-b0f00648cef8" containerName="revoke-keys" Nov 02 21:34:30.861393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.861372 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a467613-ece7-441a-a190-73a008fd332b" containerName="revoke-keys" Nov 02 21:34:30.861393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.861381 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db76fec-1d0f-448a-aecd-b0f00648cef8" containerName="revoke-keys" Nov 02 21:34:30.866611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.866586 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:30.919808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.919778 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554"] Nov 02 21:34:30.958158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.958127 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:34:30.959092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.959060 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:34:30.959268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.959250 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:34:30.959341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:30.959267 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:34:31.039090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.039054 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.039264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.039205 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.039264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.039238 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsq8f\" (UniqueName: \"kubernetes.io/projected/5689eb08-4fdf-4f80-a4a0-d31419df2c73-kube-api-access-qsq8f\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.141385 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.141299 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.141539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.141452 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.141539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.141487 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsq8f\" (UniqueName: \"kubernetes.io/projected/5689eb08-4fdf-4f80-a4a0-d31419df2c73-kube-api-access-qsq8f\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.142076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.142033 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.143940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.143917 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.195606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.195568 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsq8f\" (UniqueName: \"kubernetes.io/projected/5689eb08-4fdf-4f80-a4a0-d31419df2c73-kube-api-access-qsq8f\") pod \"maas-api-revoke-sub-416d259f-st554\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.483849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.483760 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:31.725097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.725062 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2"] Nov 02 21:34:31.729106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.729088 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.781012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.780983 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2"] Nov 02 21:34:31.852152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.852125 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.852536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.852192 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zv8t\" (UniqueName: \"kubernetes.io/projected/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-kube-api-access-7zv8t\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.852536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.852329 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.893279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.893255 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554"] Nov 02 21:34:31.896175 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:31.896146 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5689eb08_4fdf_4f80_a4a0_d31419df2c73.slice/crio-d9b4b5dceca656d0ed170a838179c6a48701dfcc8e4cd5ca48675d0cd956026e WatchSource:0}: Error finding container d9b4b5dceca656d0ed170a838179c6a48701dfcc8e4cd5ca48675d0cd956026e: Status 404 returned error can't find the container with id d9b4b5dceca656d0ed170a838179c6a48701dfcc8e4cd5ca48675d0cd956026e Nov 02 21:34:31.953678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.953655 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.953772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.953709 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zv8t\" (UniqueName: \"kubernetes.io/projected/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-kube-api-access-7zv8t\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.953772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.953767 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.954326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.954307 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:31.955908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:31.955891 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:32.008642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.008579 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zv8t\" (UniqueName: \"kubernetes.io/projected/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-kube-api-access-7zv8t\") pod \"maas-api-revoke-sub-df5ca53d-9wzh2\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:32.043351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.043320 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:32.046018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.045997 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf"] Nov 02 21:34:32.051783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.051764 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18ad616d-825nf"] Nov 02 21:34:32.233670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.233638 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" event={"ID":"5689eb08-4fdf-4f80-a4a0-d31419df2c73","Type":"ContainerStarted","Data":"685bee53f2f7b911607f57932cb708c8daa699f06f50d1c0fc1955716e3bc784"} Nov 02 21:34:32.233810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.233696 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" event={"ID":"5689eb08-4fdf-4f80-a4a0-d31419df2c73","Type":"ContainerStarted","Data":"d9b4b5dceca656d0ed170a838179c6a48701dfcc8e4cd5ca48675d0cd956026e"} Nov 02 21:34:32.349741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.349691 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" podStartSLOduration=2.349677241 podStartE2EDuration="2.349677241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:32.345494344 +0000 UTC m=+1301.901122680" watchObservedRunningTime="2026-11-02 21:34:32.349677241 +0000 UTC m=+1301.905305575" Nov 02 21:34:32.435065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.435037 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2"] Nov 02 21:34:32.437879 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:32.437852 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd1bdfa_cf68_4bd4_830d_6f6fd6b1de51.slice/crio-791074950fe0c0daaf3102ad7d7ecacbf130b0e7cc278bc5b5c70d7908e4f8ef WatchSource:0}: Error finding container 791074950fe0c0daaf3102ad7d7ecacbf130b0e7cc278bc5b5c70d7908e4f8ef: Status 404 returned error can't find the container with id 791074950fe0c0daaf3102ad7d7ecacbf130b0e7cc278bc5b5c70d7908e4f8ef Nov 02 21:34:32.464012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.463964 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d"] Nov 02 21:34:32.470322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.470299 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.552039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.552006 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d"] Nov 02 21:34:32.564057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564028 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564171 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564278 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfrw\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-kube-api-access-6sfrw\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564312 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564543 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564616 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564682 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564706 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.564766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.564736 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.591314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.591276 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad23875-d5e3-45e2-b608-3747f5879be0" path="/var/lib/kubelet/pods/6ad23875-d5e3-45e2-b608-3747f5879be0/volumes" Nov 02 21:34:32.666103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666014 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666063 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666093 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666216 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666331 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfrw\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-kube-api-access-6sfrw\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666366 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666500 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666512 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666546 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666580 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666551 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666810 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.666948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.666922 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.673817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.673790 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.674028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.674006 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.720171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.720145 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfrw\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-kube-api-access-6sfrw\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.720292 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.720240 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.794942 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.794914 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:32.931466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:32.931402 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:33.236195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.236169 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d"] Nov 02 21:34:33.240030 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:33.239999 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79fcb05_a1eb_4999_afa5_673cc8d09e30.slice/crio-7afc353fe2cec34043fc22f5573bcdd3723601b88440ad07f4fccef026bf689e WatchSource:0}: Error finding container 7afc353fe2cec34043fc22f5573bcdd3723601b88440ad07f4fccef026bf689e: Status 404 returned error can't find the container with id 7afc353fe2cec34043fc22f5573bcdd3723601b88440ad07f4fccef026bf689e Nov 02 21:34:33.242379 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.242343 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:33.242485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.242423 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:33.242485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.242465 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:33.246352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.246329 4441 generic.go:356] "Generic (PLEG): container finished" podID="5689eb08-4fdf-4f80-a4a0-d31419df2c73" containerID="685bee53f2f7b911607f57932cb708c8daa699f06f50d1c0fc1955716e3bc784" exitCode=0 Nov 02 21:34:33.246455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.246415 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" event={"ID":"5689eb08-4fdf-4f80-a4a0-d31419df2c73","Type":"ContainerDied","Data":"685bee53f2f7b911607f57932cb708c8daa699f06f50d1c0fc1955716e3bc784"} Nov 02 21:34:33.248896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.248375 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" event={"ID":"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51","Type":"ContainerStarted","Data":"9827eaa95cdd6ce5e2ba97c599640a33c3843cea7df1c4430fd6c9c2b000649f"} Nov 02 21:34:33.248896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.248426 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" event={"ID":"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51","Type":"ContainerStarted","Data":"791074950fe0c0daaf3102ad7d7ecacbf130b0e7cc278bc5b5c70d7908e4f8ef"} Nov 02 21:34:33.468530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:33.468477 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" podStartSLOduration=2.468455647 podStartE2EDuration="2.468455647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:33.461835653 +0000 UTC m=+1303.017463991" watchObservedRunningTime="2026-11-02 21:34:33.468455647 +0000 UTC m=+1303.024083983" Nov 02 21:34:34.262757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.262721 4441 generic.go:356] "Generic (PLEG): container finished" podID="cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" containerID="9827eaa95cdd6ce5e2ba97c599640a33c3843cea7df1c4430fd6c9c2b000649f" exitCode=0 Nov 02 21:34:34.263205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.262801 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" event={"ID":"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51","Type":"ContainerDied","Data":"9827eaa95cdd6ce5e2ba97c599640a33c3843cea7df1c4430fd6c9c2b000649f"} Nov 02 21:34:34.264462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.264441 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" event={"ID":"a79fcb05-a1eb-4999-afa5-673cc8d09e30","Type":"ContainerStarted","Data":"cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79"} Nov 02 21:34:34.264540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.264488 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" event={"ID":"a79fcb05-a1eb-4999-afa5-673cc8d09e30","Type":"ContainerStarted","Data":"7afc353fe2cec34043fc22f5573bcdd3723601b88440ad07f4fccef026bf689e"} Nov 02 21:34:34.520022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.519899 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podStartSLOduration=2.519879321 podStartE2EDuration="2.519879321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:34.511567309 +0000 UTC m=+1304.067195649" watchObservedRunningTime="2026-11-02 21:34:34.519879321 +0000 UTC m=+1304.075507661" Nov 02 21:34:34.546226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.546160 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:34.615622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.615584 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-openshift-ca\") pod \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " Nov 02 21:34:34.615785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.615637 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-xks-ca\") pod \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " Nov 02 21:34:34.615785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.615701 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5689eb08-4fdf-4f80-a4a0-d31419df2c73-kube-api-access-qsq8f\" (UniqueName: \"kubernetes.io/projected/5689eb08-4fdf-4f80-a4a0-d31419df2c73-kube-api-access-qsq8f\") pod \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\" (UID: \"5689eb08-4fdf-4f80-a4a0-d31419df2c73\") " Nov 02 21:34:34.615959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.615939 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-openshift-ca" pod "5689eb08-4fdf-4f80-a4a0-d31419df2c73" (UID: "5689eb08-4fdf-4f80-a4a0-d31419df2c73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:34.616378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.616333 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:34.617884 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.617855 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-xks-ca" pod "5689eb08-4fdf-4f80-a4a0-d31419df2c73" (UID: "5689eb08-4fdf-4f80-a4a0-d31419df2c73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:34.617995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.617938 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5689eb08-4fdf-4f80-a4a0-d31419df2c73-kube-api-access-qsq8f" pod "5689eb08-4fdf-4f80-a4a0-d31419df2c73" (UID: "5689eb08-4fdf-4f80-a4a0-d31419df2c73"). InnerVolumeSpecName "kube-api-access-qsq8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:34.718054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.718016 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689eb08-4fdf-4f80-a4a0-d31419df2c73-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:34.718054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.718055 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsq8f\" (UniqueName: \"kubernetes.io/projected/5689eb08-4fdf-4f80-a4a0-d31419df2c73-kube-api-access-qsq8f\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:34.796455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.796322 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:34.800114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.800072 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:34:34.800265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:34.800126 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 21:34:35.299618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.299029 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" event={"ID":"5689eb08-4fdf-4f80-a4a0-d31419df2c73","Type":"ContainerDied","Data":"d9b4b5dceca656d0ed170a838179c6a48701dfcc8e4cd5ca48675d0cd956026e"} Nov 02 21:34:35.299618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.299095 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b4b5dceca656d0ed170a838179c6a48701dfcc8e4cd5ca48675d0cd956026e" Nov 02 21:34:35.299618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.299339 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554" Nov 02 21:34:35.428020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.427988 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv"] Nov 02 21:34:35.428919 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.428902 4441 cpu_manager.go:414] "containerMap: removing container" podUID="5689eb08-4fdf-4f80-a4a0-d31419df2c73" containerName="revoke-keys" Nov 02 21:34:35.429001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.428922 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5689eb08-4fdf-4f80-a4a0-d31419df2c73" containerName="revoke-keys" Nov 02 21:34:35.429093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.429080 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5689eb08-4fdf-4f80-a4a0-d31419df2c73" containerName="revoke-keys" Nov 02 21:34:35.433781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.433754 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.462716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.462682 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv"] Nov 02 21:34:35.530582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.530506 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxmn\" (UniqueName: \"kubernetes.io/projected/e4ebfb50-c8eb-4632-b8ff-57d42415e459-kube-api-access-jvxmn\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.530735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.530645 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.530735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.530691 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.531704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.531682 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:35.647839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.647796 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-kube-api-access-7zv8t\" (UniqueName: \"kubernetes.io/projected/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-kube-api-access-7zv8t\") pod \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " Nov 02 21:34:35.648052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.647916 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-openshift-ca\") pod \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " Nov 02 21:34:35.648052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.648037 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-xks-ca\") pod \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\" (UID: \"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51\") " Nov 02 21:34:35.648741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.648414 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-openshift-ca" pod "cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" (UID: "cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:35.648741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.648441 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.648741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.648504 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.648994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.648749 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxmn\" (UniqueName: \"kubernetes.io/projected/e4ebfb50-c8eb-4632-b8ff-57d42415e459-kube-api-access-jvxmn\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.648994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.648822 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:35.664537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.656117 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-xks-ca" pod "cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" (UID: "cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:35.664537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.658201 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.664537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.658570 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.718257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.718211 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-kube-api-access-7zv8t" pod "cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" (UID: "cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51"). InnerVolumeSpecName "kube-api-access-7zv8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:35.742853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.742821 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxmn\" (UniqueName: \"kubernetes.io/projected/e4ebfb50-c8eb-4632-b8ff-57d42415e459-kube-api-access-jvxmn\") pod \"maas-api-revoke-sub-1c0d14d3-r2nsv\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.750661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.750616 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zv8t\" (UniqueName: \"kubernetes.io/projected/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-kube-api-access-7zv8t\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:35.750863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.750845 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:35.762992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.751089 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:35.803246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.800425 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:34:35.803246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:35.800487 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 21:34:36.590585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:36.590556 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" Nov 02 21:34:36.682264 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:36.682227 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ebfb50_c8eb_4632_b8ff_57d42415e459.slice/crio-11bae9b568164dcb023b8749b7f526fdf0bb9fc9512799ebd61fc2eefbf5f1f0 WatchSource:0}: Error finding container 11bae9b568164dcb023b8749b7f526fdf0bb9fc9512799ebd61fc2eefbf5f1f0: Status 404 returned error can't find the container with id 11bae9b568164dcb023b8749b7f526fdf0bb9fc9512799ebd61fc2eefbf5f1f0 Nov 02 21:34:36.799002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:36.773926 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2" event={"ID":"cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51","Type":"ContainerDied","Data":"791074950fe0c0daaf3102ad7d7ecacbf130b0e7cc278bc5b5c70d7908e4f8ef"} Nov 02 21:34:36.799002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:36.774062 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="791074950fe0c0daaf3102ad7d7ecacbf130b0e7cc278bc5b5c70d7908e4f8ef" Nov 02 21:34:36.799002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:36.774089 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv"] Nov 02 21:34:36.799002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:36.798547 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:34:36.799002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:36.798598 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 21:34:37.148090 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:34:37.148055 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd1bdfa_cf68_4bd4_830d_6f6fd6b1de51.slice\": RecentStats: unable to find data in memory cache]" Nov 02 21:34:37.369234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.367082 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:37.369234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.367564 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 21:34:37.380452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.377052 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:37.381488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.380680 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:34:37.399346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.399291 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Nov 02 21:34:37.541955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.541916 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:37.543234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.542492 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:34:37.565645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.564827 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:37.565835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.565810 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 21:34:37.637044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.636474 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:37.661993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.659680 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" event={"ID":"e4ebfb50-c8eb-4632-b8ff-57d42415e459","Type":"ContainerStarted","Data":"6ca67c462561bae9bf3835d87258d14223d83d5ed2305dbfd9c60fa54341b67d"} Nov 02 21:34:37.661993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.659752 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" event={"ID":"e4ebfb50-c8eb-4632-b8ff-57d42415e459","Type":"ContainerStarted","Data":"11bae9b568164dcb023b8749b7f526fdf0bb9fc9512799ebd61fc2eefbf5f1f0"} Nov 02 21:34:37.684990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.680040 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:37.780071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.780008 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" podStartSLOduration=2.779987283 podStartE2EDuration="2.779987283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:37.775354081 +0000 UTC m=+1307.330982437" watchObservedRunningTime="2026-11-02 21:34:37.779987283 +0000 UTC m=+1307.335615616" Nov 02 21:34:37.796753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.796722 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:34:37.797013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:37.796984 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 21:34:38.059161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.059121 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.177992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.176206 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 02 21:34:38.241991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.224772 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.296024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.261729 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:34:38.381014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.380789 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 21:34:38.401396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.388986 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 21:34:38.401396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.390015 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.401396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.394364 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:34:38.401396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.394428 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:34:38.401396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.394467 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:34:38.444063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.444013 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.448137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.448111 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:34:38.467861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.454859 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.467861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.456716 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Nov 02 21:34:38.467861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.457730 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:34:38.478082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.478043 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.548168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.548114 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.550493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.550465 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:34:38.582925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.582882 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.608388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.582889 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": context deadline exceeded" Nov 02 21:34:38.665995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.660155 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.840370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.840296 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.887045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.885687 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:34:38.887045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.885779 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 21:34:38.912992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.887810 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:38.988080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.937098 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:34:38.988080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:38.937180 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:39.339358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.337636 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:34:39.339358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.337698 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:34:39.339358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.337734 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:34:39.339358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.337768 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:34:39.341822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.341620 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:39.341993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.341957 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:34:39.343317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.343301 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:34:39.356998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.349691 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:34:39.356998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.350125 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:39.356998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.350180 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:39.356998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.352229 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:34:39.356998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.352273 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:34:39.360362 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.360341 4441 generic.go:356] "Generic (PLEG): container finished" podID="e4ebfb50-c8eb-4632-b8ff-57d42415e459" containerID="6ca67c462561bae9bf3835d87258d14223d83d5ed2305dbfd9c60fa54341b67d" exitCode=0 Nov 02 21:34:39.365063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.365043 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" event={"ID":"e4ebfb50-c8eb-4632-b8ff-57d42415e459","Type":"ContainerDied","Data":"6ca67c462561bae9bf3835d87258d14223d83d5ed2305dbfd9c60fa54341b67d"} Nov 02 21:34:39.375862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.375844 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:34:39.382061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.382028 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 21:34:39.391266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.391239 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:39.446985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.440961 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:34:39.458122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.458085 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Nov 02 21:34:39.552336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.552286 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:34:39.591059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.567571 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:34:39.608169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.608128 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:39.619925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.619894 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:34:39.620159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.620142 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:34:39.636403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.636379 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:34:39.652520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.650234 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:34:39.654256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.654236 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:34:39.654645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.654626 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:34:39.803827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.802877 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:34:39.803827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:39.802946 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 21:34:40.613737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.612336 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg"] Nov 02 21:34:40.613737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.613444 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" containerName="revoke-keys" Nov 02 21:34:40.613737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.613464 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" containerName="revoke-keys" Nov 02 21:34:40.613737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.613689 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" containerName="revoke-keys" Nov 02 21:34:40.634515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.623365 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.646660 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:34:40.646553 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:34:40.691689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.691244 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg"] Nov 02 21:34:40.707048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.707020 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:40.735875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.735850 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ebfb50-c8eb-4632-b8ff-57d42415e459-kube-api-access-jvxmn\" (UniqueName: \"kubernetes.io/projected/e4ebfb50-c8eb-4632-b8ff-57d42415e459-kube-api-access-jvxmn\") pod \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " Nov 02 21:34:40.736038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.735886 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-xks-ca\") pod \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " Nov 02 21:34:40.736038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.735909 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-openshift-ca\") pod \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\" (UID: \"e4ebfb50-c8eb-4632-b8ff-57d42415e459\") " Nov 02 21:34:40.736164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.736122 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.736283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.736265 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftw9\" (UniqueName: \"kubernetes.io/projected/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-kube-api-access-lftw9\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.736380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.736306 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.736380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.736329 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-openshift-ca" pod "e4ebfb50-c8eb-4632-b8ff-57d42415e459" (UID: "e4ebfb50-c8eb-4632-b8ff-57d42415e459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:40.736654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.736633 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:40.738569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.738527 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-xks-ca" pod "e4ebfb50-c8eb-4632-b8ff-57d42415e459" (UID: "e4ebfb50-c8eb-4632-b8ff-57d42415e459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:40.738933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.738909 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ebfb50-c8eb-4632-b8ff-57d42415e459-kube-api-access-jvxmn" pod "e4ebfb50-c8eb-4632-b8ff-57d42415e459" (UID: "e4ebfb50-c8eb-4632-b8ff-57d42415e459"). InnerVolumeSpecName "kube-api-access-jvxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:40.796161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.796120 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:34:40.796293 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.796190 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 21:34:40.838341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.838306 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.838775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.838401 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftw9\" (UniqueName: \"kubernetes.io/projected/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-kube-api-access-lftw9\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.838775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.838474 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.838775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.838556 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ebfb50-c8eb-4632-b8ff-57d42415e459-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:40.838775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.838573 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvxmn\" (UniqueName: \"kubernetes.io/projected/e4ebfb50-c8eb-4632-b8ff-57d42415e459-kube-api-access-jvxmn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:40.839150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.839119 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.840921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.840897 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:40.892799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:40.892726 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftw9\" (UniqueName: \"kubernetes.io/projected/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-kube-api-access-lftw9\") pod \"maas-api-revoke-sub-f4841349-dhtsg\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:41.004488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.004443 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:41.385770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.385736 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" Nov 02 21:34:41.385983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.385769 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv" event={"ID":"e4ebfb50-c8eb-4632-b8ff-57d42415e459","Type":"ContainerDied","Data":"11bae9b568164dcb023b8749b7f526fdf0bb9fc9512799ebd61fc2eefbf5f1f0"} Nov 02 21:34:41.385983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.385855 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11bae9b568164dcb023b8749b7f526fdf0bb9fc9512799ebd61fc2eefbf5f1f0" Nov 02 21:34:41.413888 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:41.413222 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c4c94d_9929_4db3_b7b2_3c1031cb75ef.slice/crio-c362afeb604e440a0d974e85ed9d47d0ff828e280bfbb6b472d8b768c67cb3ec WatchSource:0}: Error finding container c362afeb604e440a0d974e85ed9d47d0ff828e280bfbb6b472d8b768c67cb3ec: Status 404 returned error can't find the container with id c362afeb604e440a0d974e85ed9d47d0ff828e280bfbb6b472d8b768c67cb3ec Nov 02 21:34:41.416903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.415509 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg"] Nov 02 21:34:41.672057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.671987 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm"] Nov 02 21:34:41.672695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.672675 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e4ebfb50-c8eb-4632-b8ff-57d42415e459" containerName="revoke-keys" Nov 02 21:34:41.672695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.672695 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ebfb50-c8eb-4632-b8ff-57d42415e459" containerName="revoke-keys" Nov 02 21:34:41.672850 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.672812 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ebfb50-c8eb-4632-b8ff-57d42415e459" containerName="revoke-keys" Nov 02 21:34:41.677905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.677885 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.738991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.738550 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm"] Nov 02 21:34:41.754739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.753957 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.754739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.754032 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dxm4\" (UniqueName: \"kubernetes.io/projected/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-kube-api-access-4dxm4\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.754739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.754233 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.855192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.855156 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.855679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.855277 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.855679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.855321 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dxm4\" (UniqueName: \"kubernetes.io/projected/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-kube-api-access-4dxm4\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.856098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.856070 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.857713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.857686 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.912083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.910103 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dxm4\" (UniqueName: \"kubernetes.io/projected/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-kube-api-access-4dxm4\") pod \"maas-api-revoke-sub-1850e5f8-pmftm\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:41.958744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.958666 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765"] Nov 02 21:34:41.963160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.963131 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:41.995855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:41.995825 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:42.023291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.023255 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765"] Nov 02 21:34:42.058761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.058162 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.058761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.058420 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.058761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.058470 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llr9m\" (UniqueName: \"kubernetes.io/projected/c9a6be75-ac93-474c-b41a-8234bb782416-kube-api-access-llr9m\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.164020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.160419 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llr9m\" (UniqueName: \"kubernetes.io/projected/c9a6be75-ac93-474c-b41a-8234bb782416-kube-api-access-llr9m\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.164020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.160570 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.164020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.160842 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.164020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.161795 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.164020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.163917 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.215165 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.215064 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llr9m\" (UniqueName: \"kubernetes.io/projected/c9a6be75-ac93-474c-b41a-8234bb782416-kube-api-access-llr9m\") pod \"maas-api-revoke-sub-8e3a3d2f-fs765\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.280708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.280672 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:42.403553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.402242 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" event={"ID":"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef","Type":"ContainerStarted","Data":"000ad8f7ba3267f3e34ef0cdfb102f45774a8666b64f5a86077a5ca9e505afc8"} Nov 02 21:34:42.403553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.402339 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" event={"ID":"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef","Type":"ContainerStarted","Data":"c362afeb604e440a0d974e85ed9d47d0ff828e280bfbb6b472d8b768c67cb3ec"} Nov 02 21:34:42.422119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.422086 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm"] Nov 02 21:34:42.422279 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:42.422248 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode483842a_c8b7_4513_9e5f_12a8e6c8cafd.slice/crio-5f8658462ab422902d573cf4d815de7659187b60b81ba0f5da1aa2006a12eec1 WatchSource:0}: Error finding container 5f8658462ab422902d573cf4d815de7659187b60b81ba0f5da1aa2006a12eec1: Status 404 returned error can't find the container with id 5f8658462ab422902d573cf4d815de7659187b60b81ba0f5da1aa2006a12eec1 Nov 02 21:34:42.515278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.515218 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" podStartSLOduration=2.515199104 podStartE2EDuration="2.515199104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:42.512336097 +0000 UTC m=+1312.067964432" watchObservedRunningTime="2026-11-02 21:34:42.515199104 +0000 UTC m=+1312.070827447" Nov 02 21:34:42.673225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.673193 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765"] Nov 02 21:34:42.797133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.797088 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:34:42.797259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:42.797145 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:43.047389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.047354 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7"] Nov 02 21:34:43.052510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.052484 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ff4cc5-5qrq7"] Nov 02 21:34:43.421538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.420767 4441 generic.go:356] "Generic (PLEG): container finished" podID="b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" containerID="000ad8f7ba3267f3e34ef0cdfb102f45774a8666b64f5a86077a5ca9e505afc8" exitCode=0 Nov 02 21:34:43.421736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.421622 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" event={"ID":"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef","Type":"ContainerDied","Data":"000ad8f7ba3267f3e34ef0cdfb102f45774a8666b64f5a86077a5ca9e505afc8"} Nov 02 21:34:43.423773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.423740 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" event={"ID":"e483842a-c8b7-4513-9e5f-12a8e6c8cafd","Type":"ContainerStarted","Data":"2d569643198c189784a6204d662f6256ac7a237fff8c2d9fdfe8d232bd3e8fe1"} Nov 02 21:34:43.423917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.423808 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" event={"ID":"e483842a-c8b7-4513-9e5f-12a8e6c8cafd","Type":"ContainerStarted","Data":"5f8658462ab422902d573cf4d815de7659187b60b81ba0f5da1aa2006a12eec1"} Nov 02 21:34:43.426074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.426038 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" event={"ID":"c9a6be75-ac93-474c-b41a-8234bb782416","Type":"ContainerStarted","Data":"f7e4280df6bee81e9b82c73cc9ecabb20345881ab35413aef3c8efcf2584cd34"} Nov 02 21:34:43.426185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.426092 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" event={"ID":"c9a6be75-ac93-474c-b41a-8234bb782416","Type":"ContainerStarted","Data":"304bfe114b93fc18ea77c9bec468e6fab9a42bd3529ba04ca2aa42c398826390"} Nov 02 21:34:43.561041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.561003 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:43.562318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.562291 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:43.563018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.562958 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:34:43.629988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.629398 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt"] Nov 02 21:34:43.645926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.645890 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.647193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.647135 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" podStartSLOduration=2.6471196 podStartE2EDuration="2.6471196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:43.644162461 +0000 UTC m=+1313.199790799" watchObservedRunningTime="2026-11-02 21:34:43.6471196 +0000 UTC m=+1313.202747940" Nov 02 21:34:43.699264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.685435 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9h4w\" (UniqueName: \"kubernetes.io/projected/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-kube-api-access-b9h4w\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.699264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.685506 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.699264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.685743 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.699264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.694179 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt"] Nov 02 21:34:43.754127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.754075 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" podStartSLOduration=2.7540627779999998 podStartE2EDuration="2.754062778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:43.752601555 +0000 UTC m=+1313.308229891" watchObservedRunningTime="2026-11-02 21:34:43.754062778 +0000 UTC m=+1313.309691114" Nov 02 21:34:43.787498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.787455 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9h4w\" (UniqueName: \"kubernetes.io/projected/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-kube-api-access-b9h4w\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.787667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.787551 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.787729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.787661 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.788225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.788202 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.789914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.789892 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.842753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.842729 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9h4w\" (UniqueName: \"kubernetes.io/projected/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-kube-api-access-b9h4w\") pod \"maas-api-revoke-sub-a4e3d1fa-hcpmt\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.962282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.962206 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:43.990930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:43.990898 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:44.348236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.348211 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt"] Nov 02 21:34:44.351437 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:44.351410 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b242b9f_c0d6_49d2_9d78_a0efbc37385e.slice/crio-6770eb48e317059aed72f8f98d33c23bc941eb02373a47e1cbf9e404b6944e13 WatchSource:0}: Error finding container 6770eb48e317059aed72f8f98d33c23bc941eb02373a47e1cbf9e404b6944e13: Status 404 returned error can't find the container with id 6770eb48e317059aed72f8f98d33c23bc941eb02373a47e1cbf9e404b6944e13 Nov 02 21:34:44.437242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.437207 4441 generic.go:356] "Generic (PLEG): container finished" podID="e483842a-c8b7-4513-9e5f-12a8e6c8cafd" containerID="2d569643198c189784a6204d662f6256ac7a237fff8c2d9fdfe8d232bd3e8fe1" exitCode=0 Nov 02 21:34:44.437369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.437289 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" event={"ID":"e483842a-c8b7-4513-9e5f-12a8e6c8cafd","Type":"ContainerDied","Data":"2d569643198c189784a6204d662f6256ac7a237fff8c2d9fdfe8d232bd3e8fe1"} Nov 02 21:34:44.439027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.439007 4441 generic.go:356] "Generic (PLEG): container finished" podID="c9a6be75-ac93-474c-b41a-8234bb782416" containerID="f7e4280df6bee81e9b82c73cc9ecabb20345881ab35413aef3c8efcf2584cd34" exitCode=0 Nov 02 21:34:44.439146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.439048 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" event={"ID":"c9a6be75-ac93-474c-b41a-8234bb782416","Type":"ContainerDied","Data":"f7e4280df6bee81e9b82c73cc9ecabb20345881ab35413aef3c8efcf2584cd34"} Nov 02 21:34:44.440371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.440351 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" event={"ID":"1b242b9f-c0d6-49d2-9d78-a0efbc37385e","Type":"ContainerStarted","Data":"6770eb48e317059aed72f8f98d33c23bc941eb02373a47e1cbf9e404b6944e13"} Nov 02 21:34:44.591384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.591348 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c986c51-4b88-4483-b10f-a5c30e55f32b" path="/var/lib/kubelet/pods/1c986c51-4b88-4483-b10f-a5c30e55f32b/volumes" Nov 02 21:34:44.597120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.597092 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:44.729183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.729153 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-openshift-ca\") pod \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " Nov 02 21:34:44.729338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.729246 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-kube-api-access-lftw9\" (UniqueName: \"kubernetes.io/projected/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-kube-api-access-lftw9\") pod \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " Nov 02 21:34:44.729381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.729332 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-xks-ca\") pod \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\" (UID: \"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef\") " Nov 02 21:34:44.729576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.729551 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-openshift-ca" pod "b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" (UID: "b8c4c94d-9929-4db3-b7b2-3c1031cb75ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:44.730006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.729954 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:44.731309 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.731289 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-kube-api-access-lftw9" pod "b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" (UID: "b8c4c94d-9929-4db3-b7b2-3c1031cb75ef"). InnerVolumeSpecName "kube-api-access-lftw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:44.731405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.731364 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-xks-ca" pod "b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" (UID: "b8c4c94d-9929-4db3-b7b2-3c1031cb75ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:44.831900 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.831863 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lftw9\" (UniqueName: \"kubernetes.io/projected/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-kube-api-access-lftw9\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:44.831900 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:44.831891 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:45.046503 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.046423 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9"] Nov 02 21:34:45.051519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.051490 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0bcc74a-bhgp9"] Nov 02 21:34:45.451154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.451122 4441 generic.go:356] "Generic (PLEG): container finished" podID="1b242b9f-c0d6-49d2-9d78-a0efbc37385e" containerID="0a46a6f67bee898220565b72fbc92599e8ea07c035f508df08420550566e166f" exitCode=0 Nov 02 21:34:45.451610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.451214 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" event={"ID":"1b242b9f-c0d6-49d2-9d78-a0efbc37385e","Type":"ContainerDied","Data":"0a46a6f67bee898220565b72fbc92599e8ea07c035f508df08420550566e166f"} Nov 02 21:34:45.452937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.452915 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" Nov 02 21:34:45.453087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.452945 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg" event={"ID":"b8c4c94d-9929-4db3-b7b2-3c1031cb75ef","Type":"ContainerDied","Data":"c362afeb604e440a0d974e85ed9d47d0ff828e280bfbb6b472d8b768c67cb3ec"} Nov 02 21:34:45.453087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.453014 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c362afeb604e440a0d974e85ed9d47d0ff828e280bfbb6b472d8b768c67cb3ec" Nov 02 21:34:45.644040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.644008 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:45.667231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.667210 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:45.745275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745247 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a6be75-ac93-474c-b41a-8234bb782416-kube-api-access-llr9m\" (UniqueName: \"kubernetes.io/projected/c9a6be75-ac93-474c-b41a-8234bb782416-kube-api-access-llr9m\") pod \"c9a6be75-ac93-474c-b41a-8234bb782416\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " Nov 02 21:34:45.745466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745450 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-kube-api-access-4dxm4\" (UniqueName: \"kubernetes.io/projected/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-kube-api-access-4dxm4\") pod \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " Nov 02 21:34:45.745513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745500 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-openshift-ca\") pod \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " Nov 02 21:34:45.745578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745563 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-openshift-ca\") pod \"c9a6be75-ac93-474c-b41a-8234bb782416\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " Nov 02 21:34:45.745636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745624 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-xks-ca\") pod \"c9a6be75-ac93-474c-b41a-8234bb782416\" (UID: \"c9a6be75-ac93-474c-b41a-8234bb782416\") " Nov 02 21:34:45.745740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745721 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-xks-ca\") pod \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\" (UID: \"e483842a-c8b7-4513-9e5f-12a8e6c8cafd\") " Nov 02 21:34:45.745991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745939 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-openshift-ca" pod "e483842a-c8b7-4513-9e5f-12a8e6c8cafd" (UID: "e483842a-c8b7-4513-9e5f-12a8e6c8cafd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:45.746091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.745959 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-openshift-ca" pod "c9a6be75-ac93-474c-b41a-8234bb782416" (UID: "c9a6be75-ac93-474c-b41a-8234bb782416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:45.746294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.746270 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:45.746410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.746295 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:45.747568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.747541 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a6be75-ac93-474c-b41a-8234bb782416-kube-api-access-llr9m" pod "c9a6be75-ac93-474c-b41a-8234bb782416" (UID: "c9a6be75-ac93-474c-b41a-8234bb782416"). InnerVolumeSpecName "kube-api-access-llr9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:45.747666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.747623 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-kube-api-access-4dxm4" pod "e483842a-c8b7-4513-9e5f-12a8e6c8cafd" (UID: "e483842a-c8b7-4513-9e5f-12a8e6c8cafd"). InnerVolumeSpecName "kube-api-access-4dxm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:45.747777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.747758 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-xks-ca" pod "e483842a-c8b7-4513-9e5f-12a8e6c8cafd" (UID: "e483842a-c8b7-4513-9e5f-12a8e6c8cafd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:45.747874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.747859 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-xks-ca" pod "c9a6be75-ac93-474c-b41a-8234bb782416" (UID: "c9a6be75-ac93-474c-b41a-8234bb782416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:45.847873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.847843 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dxm4\" (UniqueName: \"kubernetes.io/projected/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-kube-api-access-4dxm4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:45.847873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.847872 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6be75-ac93-474c-b41a-8234bb782416-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:45.848077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.847885 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e483842a-c8b7-4513-9e5f-12a8e6c8cafd-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:45.848077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:45.847894 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llr9m\" (UniqueName: \"kubernetes.io/projected/c9a6be75-ac93-474c-b41a-8234bb782416-kube-api-access-llr9m\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:46.463375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.463341 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" Nov 02 21:34:46.463375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.463359 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm" event={"ID":"e483842a-c8b7-4513-9e5f-12a8e6c8cafd","Type":"ContainerDied","Data":"5f8658462ab422902d573cf4d815de7659187b60b81ba0f5da1aa2006a12eec1"} Nov 02 21:34:46.463845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.463448 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f8658462ab422902d573cf4d815de7659187b60b81ba0f5da1aa2006a12eec1" Nov 02 21:34:46.465030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.465003 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" event={"ID":"c9a6be75-ac93-474c-b41a-8234bb782416","Type":"ContainerDied","Data":"304bfe114b93fc18ea77c9bec468e6fab9a42bd3529ba04ca2aa42c398826390"} Nov 02 21:34:46.465143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.465047 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="304bfe114b93fc18ea77c9bec468e6fab9a42bd3529ba04ca2aa42c398826390" Nov 02 21:34:46.465143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.465137 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765" Nov 02 21:34:46.600571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.600506 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c658aee1-1ef6-4bcf-8430-c9922bc18c37" path="/var/lib/kubelet/pods/c658aee1-1ef6-4bcf-8430-c9922bc18c37/volumes" Nov 02 21:34:46.670297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.670269 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:46.773283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.773218 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-xks-ca\") pod \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " Nov 02 21:34:46.773283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.773252 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-kube-api-access-b9h4w\" (UniqueName: \"kubernetes.io/projected/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-kube-api-access-b9h4w\") pod \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " Nov 02 21:34:46.773465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.773354 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-openshift-ca\") pod \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\" (UID: \"1b242b9f-c0d6-49d2-9d78-a0efbc37385e\") " Nov 02 21:34:46.773784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.773756 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-openshift-ca" pod "1b242b9f-c0d6-49d2-9d78-a0efbc37385e" (UID: "1b242b9f-c0d6-49d2-9d78-a0efbc37385e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:46.774361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.774135 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:46.775500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.775474 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-xks-ca" pod "1b242b9f-c0d6-49d2-9d78-a0efbc37385e" (UID: "1b242b9f-c0d6-49d2-9d78-a0efbc37385e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:46.775500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.775490 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-kube-api-access-b9h4w" pod "1b242b9f-c0d6-49d2-9d78-a0efbc37385e" (UID: "1b242b9f-c0d6-49d2-9d78-a0efbc37385e"). InnerVolumeSpecName "kube-api-access-b9h4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:46.875786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.875753 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:46.875786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:46.875781 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9h4w\" (UniqueName: \"kubernetes.io/projected/1b242b9f-c0d6-49d2-9d78-a0efbc37385e-kube-api-access-b9h4w\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:47.046616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.046530 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv"] Nov 02 21:34:47.051448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.051419 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c680b95e-rqdzv"] Nov 02 21:34:47.201024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.200988 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk"] Nov 02 21:34:47.201557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201542 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e483842a-c8b7-4513-9e5f-12a8e6c8cafd" containerName="revoke-keys" Nov 02 21:34:47.201606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201558 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e483842a-c8b7-4513-9e5f-12a8e6c8cafd" containerName="revoke-keys" Nov 02 21:34:47.201606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201575 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c9a6be75-ac93-474c-b41a-8234bb782416" containerName="revoke-keys" Nov 02 21:34:47.201606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201580 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a6be75-ac93-474c-b41a-8234bb782416" containerName="revoke-keys" Nov 02 21:34:47.201606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201597 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" containerName="revoke-keys" Nov 02 21:34:47.201606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201602 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" containerName="revoke-keys" Nov 02 21:34:47.201753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201623 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1b242b9f-c0d6-49d2-9d78-a0efbc37385e" containerName="revoke-keys" Nov 02 21:34:47.201753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201628 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b242b9f-c0d6-49d2-9d78-a0efbc37385e" containerName="revoke-keys" Nov 02 21:34:47.201753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201716 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" containerName="revoke-keys" Nov 02 21:34:47.201753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201728 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e483842a-c8b7-4513-9e5f-12a8e6c8cafd" containerName="revoke-keys" Nov 02 21:34:47.201753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201736 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a6be75-ac93-474c-b41a-8234bb782416" containerName="revoke-keys" Nov 02 21:34:47.201753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.201744 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b242b9f-c0d6-49d2-9d78-a0efbc37385e" containerName="revoke-keys" Nov 02 21:34:47.205512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.205489 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.258005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.257977 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk"] Nov 02 21:34:47.382742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.382704 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.382979 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.382950 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqcf2\" (UniqueName: \"kubernetes.io/projected/e519b02e-deed-4936-b1e5-545668bafb67-kube-api-access-mqcf2\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.383104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.383078 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.477099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.477065 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" Nov 02 21:34:47.477099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.477083 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt" event={"ID":"1b242b9f-c0d6-49d2-9d78-a0efbc37385e","Type":"ContainerDied","Data":"6770eb48e317059aed72f8f98d33c23bc941eb02373a47e1cbf9e404b6944e13"} Nov 02 21:34:47.477579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.477136 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6770eb48e317059aed72f8f98d33c23bc941eb02373a47e1cbf9e404b6944e13" Nov 02 21:34:47.483940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.483920 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqcf2\" (UniqueName: \"kubernetes.io/projected/e519b02e-deed-4936-b1e5-545668bafb67-kube-api-access-mqcf2\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.484087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.483958 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.484087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.484042 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.484725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.484706 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.486364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.486344 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.538062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.538035 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqcf2\" (UniqueName: \"kubernetes.io/projected/e519b02e-deed-4936-b1e5-545668bafb67-kube-api-access-mqcf2\") pod \"maas-api-revoke-sub-569c8718-dkzwk\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:47.794121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.794043 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:47.820532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:47.820504 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:48.211087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.211054 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk"] Nov 02 21:34:48.214319 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:48.214291 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode519b02e_deed_4936_b1e5_545668bafb67.slice/crio-4aebe53f05e052b10860c285fb12eea4956974418a1021417b2f0451d9a506ec WatchSource:0}: Error finding container 4aebe53f05e052b10860c285fb12eea4956974418a1021417b2f0451d9a506ec: Status 404 returned error can't find the container with id 4aebe53f05e052b10860c285fb12eea4956974418a1021417b2f0451d9a506ec Nov 02 21:34:48.488748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.488721 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" event={"ID":"e519b02e-deed-4936-b1e5-545668bafb67","Type":"ContainerStarted","Data":"4aebe53f05e052b10860c285fb12eea4956974418a1021417b2f0451d9a506ec"} Nov 02 21:34:48.502587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.502562 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:48.588755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.588717 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224635dd-b127-4f40-a895-3cc532404a4f" path="/var/lib/kubelet/pods/224635dd-b127-4f40-a895-3cc532404a4f/volumes" Nov 02 21:34:48.730733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.730652 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7"] Nov 02 21:34:48.735375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.735356 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802070 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802106 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802137 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802210 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802242 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb843693-22ff-4e1b-8048-cb2b971339b4-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802261 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802353 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802371 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnj7p\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-kube-api-access-cnj7p\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.802436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.802398 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.841313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.841274 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7"] Nov 02 21:34:48.903534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903503 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.903724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903543 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.903724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903578 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.903724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903612 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.903724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903637 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb843693-22ff-4e1b-8048-cb2b971339b4-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.903724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903654 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.903724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903715 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.904075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903733 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnj7p\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-kube-api-access-cnj7p\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.904075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903778 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.904075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903949 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.904075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.903962 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.904306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.904279 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.904378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.904347 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.904778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.904754 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb843693-22ff-4e1b-8048-cb2b971339b4-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.905942 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.905920 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.906445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.906420 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.958092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.958062 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:48.958349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:48.958329 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnj7p\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-kube-api-access-cnj7p\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:49.050028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.049934 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:49.060833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.060810 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z"] Nov 02 21:34:49.068244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.068155 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811430bc-m9m7z"] Nov 02 21:34:49.391409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.391379 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:49.488840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.488810 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7"] Nov 02 21:34:49.492430 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:49.492395 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb843693_22ff_4e1b_8048_cb2b971339b4.slice/crio-d96e424fdb33e965723d6f67d57e2b20ef00b04fd908dd645641bb6ec1c86bc4 WatchSource:0}: Error finding container d96e424fdb33e965723d6f67d57e2b20ef00b04fd908dd645641bb6ec1c86bc4: Status 404 returned error can't find the container with id d96e424fdb33e965723d6f67d57e2b20ef00b04fd908dd645641bb6ec1c86bc4 Nov 02 21:34:49.494866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.494834 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:49.494959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.494920 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:49.494959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.494951 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:34:49.499024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.499002 4441 generic.go:356] "Generic (PLEG): container finished" podID="e519b02e-deed-4936-b1e5-545668bafb67" containerID="3c439a2145120fb385146aa3f7c6f53534b78ce6dc3f14a76c73d617d92bef40" exitCode=0 Nov 02 21:34:49.499114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:49.499086 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" event={"ID":"e519b02e-deed-4936-b1e5-545668bafb67","Type":"ContainerDied","Data":"3c439a2145120fb385146aa3f7c6f53534b78ce6dc3f14a76c73d617d92bef40"} Nov 02 21:34:50.510866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.510829 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" event={"ID":"cb843693-22ff-4e1b-8048-cb2b971339b4","Type":"ContainerStarted","Data":"014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb"} Nov 02 21:34:50.511351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.510883 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" event={"ID":"cb843693-22ff-4e1b-8048-cb2b971339b4","Type":"ContainerStarted","Data":"d96e424fdb33e965723d6f67d57e2b20ef00b04fd908dd645641bb6ec1c86bc4"} Nov 02 21:34:50.630157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.628006 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7cb0869-4634-46af-b380-c1d04cae724a" path="/var/lib/kubelet/pods/b7cb0869-4634-46af-b380-c1d04cae724a/volumes" Nov 02 21:34:50.764119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.764057 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:50.843074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.843042 4441 scope.go:122] "RemoveContainer" containerID="38043e4c8631c6d334d2ac146c1262383e8f97875f94acda63f494d14b2dd187" Nov 02 21:34:50.852115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.852071 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podStartSLOduration=2.852055846 podStartE2EDuration="2.852055846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:50.647114628 +0000 UTC m=+1320.202742965" watchObservedRunningTime="2026-11-02 21:34:50.852055846 +0000 UTC m=+1320.407684200" Nov 02 21:34:50.860790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.860748 4441 scope.go:122] "RemoveContainer" containerID="bade7190ef34ee275a4cf6f7424d1763e183ece0c4e326cedca7bbd851a4880d" Nov 02 21:34:50.862120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.862098 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-openshift-ca\") pod \"e519b02e-deed-4936-b1e5-545668bafb67\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " Nov 02 21:34:50.862185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.862157 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-xks-ca\") pod \"e519b02e-deed-4936-b1e5-545668bafb67\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " Nov 02 21:34:50.862259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.862242 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e519b02e-deed-4936-b1e5-545668bafb67-kube-api-access-mqcf2\" (UniqueName: \"kubernetes.io/projected/e519b02e-deed-4936-b1e5-545668bafb67-kube-api-access-mqcf2\") pod \"e519b02e-deed-4936-b1e5-545668bafb67\" (UID: \"e519b02e-deed-4936-b1e5-545668bafb67\") " Nov 02 21:34:50.862873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.862846 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-openshift-ca" pod "e519b02e-deed-4936-b1e5-545668bafb67" (UID: "e519b02e-deed-4936-b1e5-545668bafb67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:50.864054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.864027 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:50.866520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.866493 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e519b02e-deed-4936-b1e5-545668bafb67-kube-api-access-mqcf2" pod "e519b02e-deed-4936-b1e5-545668bafb67" (UID: "e519b02e-deed-4936-b1e5-545668bafb67"). InnerVolumeSpecName "kube-api-access-mqcf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:50.909000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.908341 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-xks-ca" pod "e519b02e-deed-4936-b1e5-545668bafb67" (UID: "e519b02e-deed-4936-b1e5-545668bafb67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:50.923302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.923269 4441 scope.go:122] "RemoveContainer" containerID="fabca8f225c236d32ad601003736bf72562df70c6944333c0d29b38a83dff932" Nov 02 21:34:50.937364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.937342 4441 scope.go:122] "RemoveContainer" containerID="c1233461bc1e90032503d93a2bc1674a1d73eff360a0638851467e547bfa7271" Nov 02 21:34:50.949038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.949016 4441 scope.go:122] "RemoveContainer" containerID="eb649dea75c7d2ec8a4c7a5b3c1390262a499f981bbacc6a40cf03ac09857537" Nov 02 21:34:50.965344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.965317 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e519b02e-deed-4936-b1e5-545668bafb67-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:50.965464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:50.965349 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqcf2\" (UniqueName: \"kubernetes.io/projected/e519b02e-deed-4936-b1e5-545668bafb67-kube-api-access-mqcf2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:51.051142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.051113 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:51.052688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.052663 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 21:34:51.052780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.052717 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 21:34:51.290046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.285791 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:51.421063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.421021 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:51.522577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.522546 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" Nov 02 21:34:51.522577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.522565 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk" event={"ID":"e519b02e-deed-4936-b1e5-545668bafb67","Type":"ContainerDied","Data":"4aebe53f05e052b10860c285fb12eea4956974418a1021417b2f0451d9a506ec"} Nov 02 21:34:51.523046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:51.522625 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aebe53f05e052b10860c285fb12eea4956974418a1021417b2f0451d9a506ec" Nov 02 21:34:52.050762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.050722 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 21:34:52.050913 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.050785 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 21:34:52.122010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.121963 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq"] Nov 02 21:34:52.122672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.122651 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e519b02e-deed-4936-b1e5-545668bafb67" containerName="revoke-keys" Nov 02 21:34:52.122779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.122673 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e519b02e-deed-4936-b1e5-545668bafb67" containerName="revoke-keys" Nov 02 21:34:52.122856 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.122844 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e519b02e-deed-4936-b1e5-545668bafb67" containerName="revoke-keys" Nov 02 21:34:52.128611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.128588 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.199474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.195335 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:34:52.200324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.200301 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq"] Nov 02 21:34:52.203762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.203705 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:34:52.203762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.203699 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:34:52.203762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.203710 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:34:52.208820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.208790 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.209020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.208980 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxdk\" (UniqueName: \"kubernetes.io/projected/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-kube-api-access-mrxdk\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.209107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.209085 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.310830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.310798 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxdk\" (UniqueName: \"kubernetes.io/projected/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-kube-api-access-mrxdk\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.311038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.310858 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.311111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.311064 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.311665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.311640 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.313366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.313339 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.365276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.365249 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxdk\" (UniqueName: \"kubernetes.io/projected/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-kube-api-access-mrxdk\") pod \"maas-api-revoke-sub-73e9ab4d-pbmxq\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.445746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.445716 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:52.864936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:52.864897 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq"] Nov 02 21:34:52.867194 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:52.867168 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e86e58_ea15_4ea1_8b7a_3f2929d0a318.slice/crio-2d609f7d543689f010c1d49aa139194719e20e0b9837d6edadadd0ab67881f95 WatchSource:0}: Error finding container 2d609f7d543689f010c1d49aa139194719e20e0b9837d6edadadd0ab67881f95: Status 404 returned error can't find the container with id 2d609f7d543689f010c1d49aa139194719e20e0b9837d6edadadd0ab67881f95 Nov 02 21:34:53.048210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.048174 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm"] Nov 02 21:34:53.050507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.050481 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 21:34:53.050640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.050526 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 21:34:53.053252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.053232 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50154a20-zgvrm"] Nov 02 21:34:53.122296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.122228 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:53.545416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.545326 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" event={"ID":"96e86e58-ea15-4ea1-8b7a-3f2929d0a318","Type":"ContainerStarted","Data":"494ce1f380fa07c9695548fc643d6f1b96137b52a3117bd391cb8f2610919aab"} Nov 02 21:34:53.545416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.545384 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" event={"ID":"96e86e58-ea15-4ea1-8b7a-3f2929d0a318","Type":"ContainerStarted","Data":"2d609f7d543689f010c1d49aa139194719e20e0b9837d6edadadd0ab67881f95"} Nov 02 21:34:53.657624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:53.657568 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" podStartSLOduration=1.657549776 podStartE2EDuration="1.657549776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:53.65689844 +0000 UTC m=+1323.212526768" watchObservedRunningTime="2026-11-02 21:34:53.657549776 +0000 UTC m=+1323.213178113" Nov 02 21:34:54.047237 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.047204 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c"] Nov 02 21:34:54.051305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.051277 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 21:34:54.051430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.051327 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 21:34:54.052261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.052239 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03d77d8a-9jt2c"] Nov 02 21:34:54.558048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.558006 4441 generic.go:356] "Generic (PLEG): container finished" podID="96e86e58-ea15-4ea1-8b7a-3f2929d0a318" containerID="494ce1f380fa07c9695548fc643d6f1b96137b52a3117bd391cb8f2610919aab" exitCode=0 Nov 02 21:34:54.558228 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.558090 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" event={"ID":"96e86e58-ea15-4ea1-8b7a-3f2929d0a318","Type":"ContainerDied","Data":"494ce1f380fa07c9695548fc643d6f1b96137b52a3117bd391cb8f2610919aab"} Nov 02 21:34:54.601171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.601129 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0" path="/var/lib/kubelet/pods/a671c0cc-37dd-4edb-bfb6-8d6228d3d1c0/volumes" Nov 02 21:34:54.601888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:54.601865 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88080f4-e327-4de3-bc87-47933fe0b2f9" path="/var/lib/kubelet/pods/a88080f4-e327-4de3-bc87-47933fe0b2f9/volumes" Nov 02 21:34:55.051276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.051174 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 21:34:55.051276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.051238 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 21:34:55.770061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.770027 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:55.871691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.871612 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-openshift-ca\") pod \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " Nov 02 21:34:55.871826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.871740 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-xks-ca\") pod \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " Nov 02 21:34:55.871826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.871772 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-kube-api-access-mrxdk\" (UniqueName: \"kubernetes.io/projected/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-kube-api-access-mrxdk\") pod \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\" (UID: \"96e86e58-ea15-4ea1-8b7a-3f2929d0a318\") " Nov 02 21:34:55.871996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.871963 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-openshift-ca" pod "96e86e58-ea15-4ea1-8b7a-3f2929d0a318" (UID: "96e86e58-ea15-4ea1-8b7a-3f2929d0a318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:55.872276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.872253 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:55.874057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.874021 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-kube-api-access-mrxdk" pod "96e86e58-ea15-4ea1-8b7a-3f2929d0a318" (UID: "96e86e58-ea15-4ea1-8b7a-3f2929d0a318"). InnerVolumeSpecName "kube-api-access-mrxdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:55.874134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.874104 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-xks-ca" pod "96e86e58-ea15-4ea1-8b7a-3f2929d0a318" (UID: "96e86e58-ea15-4ea1-8b7a-3f2929d0a318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:55.931885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.931843 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:55.973662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.973633 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:55.973662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:55.973660 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxdk\" (UniqueName: \"kubernetes.io/projected/96e86e58-ea15-4ea1-8b7a-3f2929d0a318-kube-api-access-mrxdk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.050986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:56.050941 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 21:34:56.051167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:56.051024 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 21:34:56.593414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:56.593380 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" Nov 02 21:34:56.593414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:56.593404 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq" event={"ID":"96e86e58-ea15-4ea1-8b7a-3f2929d0a318","Type":"ContainerDied","Data":"2d609f7d543689f010c1d49aa139194719e20e0b9837d6edadadd0ab67881f95"} Nov 02 21:34:56.593814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:56.593450 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d609f7d543689f010c1d49aa139194719e20e0b9837d6edadadd0ab67881f95" Nov 02 21:34:57.055175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:57.055092 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:57.055480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:57.055447 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:57.056209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:57.056191 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:34:57.787311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:57.787272 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:58.391253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.391215 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg"] Nov 02 21:34:58.391846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.391828 4441 cpu_manager.go:414] "containerMap: removing container" podUID="96e86e58-ea15-4ea1-8b7a-3f2929d0a318" containerName="revoke-keys" Nov 02 21:34:58.391846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.391845 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e86e58-ea15-4ea1-8b7a-3f2929d0a318" containerName="revoke-keys" Nov 02 21:34:58.392095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.391983 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e86e58-ea15-4ea1-8b7a-3f2929d0a318" containerName="revoke-keys" Nov 02 21:34:58.395982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.395948 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.448427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.448396 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg"] Nov 02 21:34:58.488439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.488405 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:34:58.488688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.488652 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:34:58.488799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.488780 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:34:58.488901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.488548 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:34:58.513322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.513288 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.513480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.513460 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw75c\" (UniqueName: \"kubernetes.io/projected/f1f7842a-62b4-4ad8-9556-55a250cfc104-kube-api-access-lw75c\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.513768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.513744 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.615506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.615477 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.615673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.615539 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.615673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.615602 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw75c\" (UniqueName: \"kubernetes.io/projected/f1f7842a-62b4-4ad8-9556-55a250cfc104-kube-api-access-lw75c\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.616254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.616233 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.617851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.617821 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.634569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.634547 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7"] Nov 02 21:34:58.639144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.639124 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.669814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.669794 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw75c\" (UniqueName: \"kubernetes.io/projected/f1f7842a-62b4-4ad8-9556-55a250cfc104-kube-api-access-lw75c\") pod \"maas-api-revoke-sub-21292c82-qx5pg\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.691473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.691443 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7"] Nov 02 21:34:58.710946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.710917 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:34:58.716723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.716702 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.718117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.717232 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f98f\" (UniqueName: \"kubernetes.io/projected/78a46bb5-e6f6-44af-b056-0221640aa117-kube-api-access-7f98f\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.718117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.717396 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.819627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.819591 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.820062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.819834 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f98f\" (UniqueName: \"kubernetes.io/projected/78a46bb5-e6f6-44af-b056-0221640aa117-kube-api-access-7f98f\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.820062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.819894 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.820834 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.820757 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.823801 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.822876 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.878019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.877568 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f98f\" (UniqueName: \"kubernetes.io/projected/78a46bb5-e6f6-44af-b056-0221640aa117-kube-api-access-7f98f\") pod \"maas-api-revoke-sub-0b07d639-7znh7\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:58.953564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:58.953494 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:34:59.106711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:59.106685 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg"] Nov 02 21:34:59.109308 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:59.109266 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f7842a_62b4_4ad8_9556_55a250cfc104.slice/crio-383c672e280b31420c186241aefb6d5d40ca8075dac6532d7df6031c3936a1cc WatchSource:0}: Error finding container 383c672e280b31420c186241aefb6d5d40ca8075dac6532d7df6031c3936a1cc: Status 404 returned error can't find the container with id 383c672e280b31420c186241aefb6d5d40ca8075dac6532d7df6031c3936a1cc Nov 02 21:34:59.335365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:59.335336 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7"] Nov 02 21:34:59.371063 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:34:59.371031 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a46bb5_e6f6_44af_b056_0221640aa117.slice/crio-7fb464c33d8cfd7ad169ae8922a43706bfe912562ce2649a2703afa15413bc32 WatchSource:0}: Error finding container 7fb464c33d8cfd7ad169ae8922a43706bfe912562ce2649a2703afa15413bc32: Status 404 returned error can't find the container with id 7fb464c33d8cfd7ad169ae8922a43706bfe912562ce2649a2703afa15413bc32 Nov 02 21:34:59.631575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:59.631542 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" event={"ID":"78a46bb5-e6f6-44af-b056-0221640aa117","Type":"ContainerStarted","Data":"7fb464c33d8cfd7ad169ae8922a43706bfe912562ce2649a2703afa15413bc32"} Nov 02 21:34:59.633294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:59.633268 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" event={"ID":"f1f7842a-62b4-4ad8-9556-55a250cfc104","Type":"ContainerStarted","Data":"effe2c0698c59582a51b572783dd17dd01370f14cd67d85ee9aadc2ccddffb0d"} Nov 02 21:34:59.633389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:59.633308 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" event={"ID":"f1f7842a-62b4-4ad8-9556-55a250cfc104","Type":"ContainerStarted","Data":"383c672e280b31420c186241aefb6d5d40ca8075dac6532d7df6031c3936a1cc"} Nov 02 21:34:59.754013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:34:59.750300 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" podStartSLOduration=1.7502793909999999 podStartE2EDuration="1.750279391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:59.746797872 +0000 UTC m=+1329.302426240" watchObservedRunningTime="2026-11-02 21:34:59.750279391 +0000 UTC m=+1329.305907730" Nov 02 21:35:00.061138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.061100 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn"] Nov 02 21:35:00.066308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.066278 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7038caa5-qc2wn"] Nov 02 21:35:00.112301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.112271 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp"] Nov 02 21:35:00.116688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.116663 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.172417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.172382 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp"] Nov 02 21:35:00.246689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.246615 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.246892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.246868 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.246932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.246917 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtcl\" (UniqueName: \"kubernetes.io/projected/2eae43ad-a3e4-4915-9a22-beb654744c1f-kube-api-access-4qtcl\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.348357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.348324 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.348548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.348383 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtcl\" (UniqueName: \"kubernetes.io/projected/2eae43ad-a3e4-4915-9a22-beb654744c1f-kube-api-access-4qtcl\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.348548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.348501 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.349280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.349257 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.351545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.351518 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.402679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.402652 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtcl\" (UniqueName: \"kubernetes.io/projected/2eae43ad-a3e4-4915-9a22-beb654744c1f-kube-api-access-4qtcl\") pod \"maas-api-revoke-sub-c46f0864-rcrwp\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.430727 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.430691 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:00.587520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.587483 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:00.602054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.601764 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e66e75-8f28-4c0c-8a38-843bfebcca69" path="/var/lib/kubelet/pods/45e66e75-8f28-4c0c-8a38-843bfebcca69/volumes" Nov 02 21:35:00.645889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.645850 4441 generic.go:356] "Generic (PLEG): container finished" podID="78a46bb5-e6f6-44af-b056-0221640aa117" containerID="fcc4394fdff4d6aad65aab380df407151cb1f46b0b409fc0405cde2bd2613fec" exitCode=0 Nov 02 21:35:00.646112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.646004 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" event={"ID":"78a46bb5-e6f6-44af-b056-0221640aa117","Type":"ContainerDied","Data":"fcc4394fdff4d6aad65aab380df407151cb1f46b0b409fc0405cde2bd2613fec"} Nov 02 21:35:00.648286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.648258 4441 generic.go:356] "Generic (PLEG): container finished" podID="f1f7842a-62b4-4ad8-9556-55a250cfc104" containerID="effe2c0698c59582a51b572783dd17dd01370f14cd67d85ee9aadc2ccddffb0d" exitCode=0 Nov 02 21:35:00.648631 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.648425 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" event={"ID":"f1f7842a-62b4-4ad8-9556-55a250cfc104","Type":"ContainerDied","Data":"effe2c0698c59582a51b572783dd17dd01370f14cd67d85ee9aadc2ccddffb0d"} Nov 02 21:35:00.818309 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:00.818278 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp"] Nov 02 21:35:00.821135 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:00.821104 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eae43ad_a3e4_4915_9a22_beb654744c1f.slice/crio-92d2270e583f5cd23a0730c5aa8b6d0e3e0ec64fb1680938938215b36c25d3f8 WatchSource:0}: Error finding container 92d2270e583f5cd23a0730c5aa8b6d0e3e0ec64fb1680938938215b36c25d3f8: Status 404 returned error can't find the container with id 92d2270e583f5cd23a0730c5aa8b6d0e3e0ec64fb1680938938215b36c25d3f8 Nov 02 21:35:01.048859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:01.048821 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv"] Nov 02 21:35:01.054140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:01.054097 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816742d1-l4jwv"] Nov 02 21:35:01.659781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:01.659732 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" event={"ID":"2eae43ad-a3e4-4915-9a22-beb654744c1f","Type":"ContainerStarted","Data":"dca0c8d8605c6f51a7995d9055973a4e5a9e83b430f153f0d3e132bededd511c"} Nov 02 21:35:01.660389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:01.659799 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" event={"ID":"2eae43ad-a3e4-4915-9a22-beb654744c1f","Type":"ContainerStarted","Data":"92d2270e583f5cd23a0730c5aa8b6d0e3e0ec64fb1680938938215b36c25d3f8"} Nov 02 21:35:01.774230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:01.774170 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" podStartSLOduration=1.774152927 podStartE2EDuration="1.774152927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:01.766357702 +0000 UTC m=+1331.321986039" watchObservedRunningTime="2026-11-02 21:35:01.774152927 +0000 UTC m=+1331.329781265" Nov 02 21:35:01.891368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:01.891342 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:35:01.902153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:01.902128 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:35:02.073682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.073650 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a46bb5-e6f6-44af-b056-0221640aa117-kube-api-access-7f98f\" (UniqueName: \"kubernetes.io/projected/78a46bb5-e6f6-44af-b056-0221640aa117-kube-api-access-7f98f\") pod \"78a46bb5-e6f6-44af-b056-0221640aa117\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " Nov 02 21:35:02.073889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.073816 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-openshift-ca\") pod \"78a46bb5-e6f6-44af-b056-0221640aa117\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " Nov 02 21:35:02.073889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.073874 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-xks-ca\") pod \"78a46bb5-e6f6-44af-b056-0221640aa117\" (UID: \"78a46bb5-e6f6-44af-b056-0221640aa117\") " Nov 02 21:35:02.074021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.073929 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-xks-ca\") pod \"f1f7842a-62b4-4ad8-9556-55a250cfc104\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " Nov 02 21:35:02.074021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.073987 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f7842a-62b4-4ad8-9556-55a250cfc104-kube-api-access-lw75c\" (UniqueName: \"kubernetes.io/projected/f1f7842a-62b4-4ad8-9556-55a250cfc104-kube-api-access-lw75c\") pod \"f1f7842a-62b4-4ad8-9556-55a250cfc104\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " Nov 02 21:35:02.074132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.074029 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-openshift-ca\") pod \"f1f7842a-62b4-4ad8-9556-55a250cfc104\" (UID: \"f1f7842a-62b4-4ad8-9556-55a250cfc104\") " Nov 02 21:35:02.074239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.074211 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-openshift-ca" pod "78a46bb5-e6f6-44af-b056-0221640aa117" (UID: "78a46bb5-e6f6-44af-b056-0221640aa117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:02.074681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.074655 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-openshift-ca" pod "f1f7842a-62b4-4ad8-9556-55a250cfc104" (UID: "f1f7842a-62b4-4ad8-9556-55a250cfc104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:02.074897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.074858 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:02.074897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.074881 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:02.076327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.076302 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f7842a-62b4-4ad8-9556-55a250cfc104-kube-api-access-lw75c" pod "f1f7842a-62b4-4ad8-9556-55a250cfc104" (UID: "f1f7842a-62b4-4ad8-9556-55a250cfc104"). InnerVolumeSpecName "kube-api-access-lw75c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:02.076420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.076337 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-xks-ca" pod "f1f7842a-62b4-4ad8-9556-55a250cfc104" (UID: "f1f7842a-62b4-4ad8-9556-55a250cfc104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:02.076539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.076519 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-xks-ca" pod "78a46bb5-e6f6-44af-b056-0221640aa117" (UID: "78a46bb5-e6f6-44af-b056-0221640aa117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:02.076672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.076657 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a46bb5-e6f6-44af-b056-0221640aa117-kube-api-access-7f98f" pod "78a46bb5-e6f6-44af-b056-0221640aa117" (UID: "78a46bb5-e6f6-44af-b056-0221640aa117"). InnerVolumeSpecName "kube-api-access-7f98f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:02.176588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.176556 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f98f\" (UniqueName: \"kubernetes.io/projected/78a46bb5-e6f6-44af-b056-0221640aa117-kube-api-access-7f98f\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:02.176588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.176585 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a46bb5-e6f6-44af-b056-0221640aa117-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:02.176793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.176600 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f7842a-62b4-4ad8-9556-55a250cfc104-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:02.176793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.176609 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw75c\" (UniqueName: \"kubernetes.io/projected/f1f7842a-62b4-4ad8-9556-55a250cfc104-kube-api-access-lw75c\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:02.298860 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.298825 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:02.400040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.399161 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql"] Nov 02 21:35:02.400040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.399909 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f1f7842a-62b4-4ad8-9556-55a250cfc104" containerName="revoke-keys" Nov 02 21:35:02.400040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.399927 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f7842a-62b4-4ad8-9556-55a250cfc104" containerName="revoke-keys" Nov 02 21:35:02.400040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.399942 4441 cpu_manager.go:414] "containerMap: removing container" podUID="78a46bb5-e6f6-44af-b056-0221640aa117" containerName="revoke-keys" Nov 02 21:35:02.400040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.399947 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a46bb5-e6f6-44af-b056-0221640aa117" containerName="revoke-keys" Nov 02 21:35:02.400422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.400131 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f7842a-62b4-4ad8-9556-55a250cfc104" containerName="revoke-keys" Nov 02 21:35:02.400422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.400146 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a46bb5-e6f6-44af-b056-0221640aa117" containerName="revoke-keys" Nov 02 21:35:02.405712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.405685 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.472699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.472664 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql"] Nov 02 21:35:02.582554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.582519 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.582731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.582659 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.582731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.582696 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.582826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.582731 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.582867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.582830 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.582940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.582927 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.583055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.583037 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.583132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.583073 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.583193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.583146 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprdw\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-kube-api-access-pprdw\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.587069 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.587046 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34eee63-3b3b-4b59-b525-6eee4e98a05b" path="/var/lib/kubelet/pods/d34eee63-3b3b-4b59-b525-6eee4e98a05b/volumes" Nov 02 21:35:02.659727 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:35:02.659643 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a46bb5_e6f6_44af_b056_0221640aa117.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f7842a_62b4_4ad8_9556_55a250cfc104.slice\": RecentStats: unable to find data in memory cache]" Nov 02 21:35:02.671273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.671246 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" Nov 02 21:35:02.671630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.671290 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7" event={"ID":"78a46bb5-e6f6-44af-b056-0221640aa117","Type":"ContainerDied","Data":"7fb464c33d8cfd7ad169ae8922a43706bfe912562ce2649a2703afa15413bc32"} Nov 02 21:35:02.671630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.671347 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb464c33d8cfd7ad169ae8922a43706bfe912562ce2649a2703afa15413bc32" Nov 02 21:35:02.672839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.672821 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" Nov 02 21:35:02.672988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.672818 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg" event={"ID":"f1f7842a-62b4-4ad8-9556-55a250cfc104","Type":"ContainerDied","Data":"383c672e280b31420c186241aefb6d5d40ca8075dac6532d7df6031c3936a1cc"} Nov 02 21:35:02.672988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.672936 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383c672e280b31420c186241aefb6d5d40ca8075dac6532d7df6031c3936a1cc" Nov 02 21:35:02.674842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.674812 4441 generic.go:356] "Generic (PLEG): container finished" podID="2eae43ad-a3e4-4915-9a22-beb654744c1f" containerID="dca0c8d8605c6f51a7995d9055973a4e5a9e83b430f153f0d3e132bededd511c" exitCode=0 Nov 02 21:35:02.674955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.674903 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" event={"ID":"2eae43ad-a3e4-4915-9a22-beb654744c1f","Type":"ContainerDied","Data":"dca0c8d8605c6f51a7995d9055973a4e5a9e83b430f153f0d3e132bededd511c"} Nov 02 21:35:02.684901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.684878 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.684911 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685033 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685096 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685121 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685155 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprdw\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-kube-api-access-pprdw\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685225 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685293 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685445 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685488 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685677 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685727 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685794 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.685912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.685891 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.687425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.687406 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.687425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.687412 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.738588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.738563 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:02.738707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:02.738640 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprdw\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-kube-api-access-pprdw\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:03.021149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.021062 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:03.469585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.469551 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql"] Nov 02 21:35:03.492703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.492660 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:03.492785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.492749 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:03.492831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.492790 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:03.687794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.687761 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" event={"ID":"8b0ba330-cf40-4a1e-b2bb-6a18960f7260","Type":"ContainerStarted","Data":"5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d"} Nov 02 21:35:03.688413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.688386 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" event={"ID":"8b0ba330-cf40-4a1e-b2bb-6a18960f7260","Type":"ContainerStarted","Data":"ee4d885eff47c51781c9b9affbe377a434cfbb8779bc99cb34a5b21171bc8c76"} Nov 02 21:35:03.802765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.802698 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podStartSLOduration=1.802677045 podStartE2EDuration="1.802677045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:03.797020453 +0000 UTC m=+1333.352648789" watchObservedRunningTime="2026-11-02 21:35:03.802677045 +0000 UTC m=+1333.358305382" Nov 02 21:35:03.899206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.899175 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:03.910315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.910285 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-xks-ca\") pod \"2eae43ad-a3e4-4915-9a22-beb654744c1f\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " Nov 02 21:35:03.910487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.910325 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eae43ad-a3e4-4915-9a22-beb654744c1f-kube-api-access-4qtcl\" (UniqueName: \"kubernetes.io/projected/2eae43ad-a3e4-4915-9a22-beb654744c1f-kube-api-access-4qtcl\") pod \"2eae43ad-a3e4-4915-9a22-beb654744c1f\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " Nov 02 21:35:03.910487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.910463 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-openshift-ca\") pod \"2eae43ad-a3e4-4915-9a22-beb654744c1f\" (UID: \"2eae43ad-a3e4-4915-9a22-beb654744c1f\") " Nov 02 21:35:03.911139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.911096 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-openshift-ca" pod "2eae43ad-a3e4-4915-9a22-beb654744c1f" (UID: "2eae43ad-a3e4-4915-9a22-beb654744c1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:03.913385 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.913359 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eae43ad-a3e4-4915-9a22-beb654744c1f-kube-api-access-4qtcl" pod "2eae43ad-a3e4-4915-9a22-beb654744c1f" (UID: "2eae43ad-a3e4-4915-9a22-beb654744c1f"). InnerVolumeSpecName "kube-api-access-4qtcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:03.913545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:03.913527 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-xks-ca" pod "2eae43ad-a3e4-4915-9a22-beb654744c1f" (UID: "2eae43ad-a3e4-4915-9a22-beb654744c1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:04.012236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.012151 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:04.012236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.012189 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qtcl\" (UniqueName: \"kubernetes.io/projected/2eae43ad-a3e4-4915-9a22-beb654744c1f-kube-api-access-4qtcl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:04.012236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.012204 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eae43ad-a3e4-4915-9a22-beb654744c1f-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:04.021415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.021381 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:04.022859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.022831 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:35:04.023009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.022875 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 02 21:35:04.716393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.716361 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" Nov 02 21:35:04.716876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.716405 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp" event={"ID":"2eae43ad-a3e4-4915-9a22-beb654744c1f","Type":"ContainerDied","Data":"92d2270e583f5cd23a0730c5aa8b6d0e3e0ec64fb1680938938215b36c25d3f8"} Nov 02 21:35:04.716876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:04.716455 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92d2270e583f5cd23a0730c5aa8b6d0e3e0ec64fb1680938938215b36c25d3f8" Nov 02 21:35:05.021710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:05.021627 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:35:05.021710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:05.021683 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 02 21:35:06.022285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:06.022245 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:35:06.022680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:06.022314 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 02 21:35:06.382024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:06.376711 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:06.563853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:06.563810 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:06.921117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:06.921084 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:07.028477 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:07.028432 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:35:07.029126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:07.029090 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 02 21:35:07.167997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:07.167952 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:08.021456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.021420 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:35:08.021647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.021472 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 02 21:35:08.465801 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.465768 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c"] Nov 02 21:35:08.466435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.466420 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2eae43ad-a3e4-4915-9a22-beb654744c1f" containerName="revoke-keys" Nov 02 21:35:08.466497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.466437 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eae43ad-a3e4-4915-9a22-beb654744c1f" containerName="revoke-keys" Nov 02 21:35:08.466570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.466558 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eae43ad-a3e4-4915-9a22-beb654744c1f" containerName="revoke-keys" Nov 02 21:35:08.477451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.477425 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.522863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.522830 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c"] Nov 02 21:35:08.564579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.564550 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:35:08.564773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.564625 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:35:08.564864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.564668 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:35:08.583173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.583147 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:35:08.601983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.601936 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zws8v\" (UniqueName: \"kubernetes.io/projected/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-kube-api-access-zws8v\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.602260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.602240 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.602441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.602415 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.704595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.704555 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zws8v\" (UniqueName: \"kubernetes.io/projected/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-kube-api-access-zws8v\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.704787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.704624 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.704787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.704670 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.705588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.705565 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.707449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.707423 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.758599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.758529 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zws8v\" (UniqueName: \"kubernetes.io/projected/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-kube-api-access-zws8v\") pod \"maas-api-revoke-sub-2984bd0a-hbd5c\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:08.808597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:08.808561 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:09.021840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.021465 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:35:09.021840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.021510 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 02 21:35:09.115245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.115207 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx"] Nov 02 21:35:09.121846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.121816 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.206992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.189850 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx"] Nov 02 21:35:09.223870 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.223833 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.224075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.224034 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hv8g\" (UniqueName: \"kubernetes.io/projected/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-kube-api-access-4hv8g\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.224164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.224142 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.230044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.230007 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c"] Nov 02 21:35:09.234446 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:09.234418 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaedf9a3_6eae_4435_9a5c_35a36ff16a40.slice/crio-e02f0ac75032e93cec899c51600fbd5bd20beec5e85426ea984d55c7927e2a7a WatchSource:0}: Error finding container e02f0ac75032e93cec899c51600fbd5bd20beec5e85426ea984d55c7927e2a7a: Status 404 returned error can't find the container with id e02f0ac75032e93cec899c51600fbd5bd20beec5e85426ea984d55c7927e2a7a Nov 02 21:35:09.325621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.325590 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.325803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.325692 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv8g\" (UniqueName: \"kubernetes.io/projected/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-kube-api-access-4hv8g\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.325803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.325791 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.326680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.326656 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.327945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.327923 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.348603 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.348578 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj"] Nov 02 21:35:09.352770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.352753 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.414023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.413994 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj"] Nov 02 21:35:09.428060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.428026 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr49l\" (UniqueName: \"kubernetes.io/projected/56fb272b-6df7-4a87-a2d9-9447e929538f-kube-api-access-tr49l\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.428206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.428147 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.428206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.428197 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.488992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.485145 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hv8g\" (UniqueName: \"kubernetes.io/projected/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-kube-api-access-4hv8g\") pod \"maas-api-revoke-sub-a2f12db8-nw4sx\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.531500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.531417 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr49l\" (UniqueName: \"kubernetes.io/projected/56fb272b-6df7-4a87-a2d9-9447e929538f-kube-api-access-tr49l\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.531629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.531502 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.531629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.531532 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.532331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.532300 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.534213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.534173 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.590907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.590876 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr49l\" (UniqueName: \"kubernetes.io/projected/56fb272b-6df7-4a87-a2d9-9447e929538f-kube-api-access-tr49l\") pod \"maas-api-revoke-sub-71d4da39-dc6vj\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.668515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.668485 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:09.737211 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.737182 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:09.780262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:09.772028 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" event={"ID":"aaedf9a3-6eae-4435-9a5c-35a36ff16a40","Type":"ContainerStarted","Data":"e02f0ac75032e93cec899c51600fbd5bd20beec5e85426ea984d55c7927e2a7a"} Nov 02 21:35:10.022698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.022665 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:35:10.022857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.022716 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 02 21:35:10.035576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.035545 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:10.090431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.090400 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj"] Nov 02 21:35:10.091109 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:10.091071 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fb272b_6df7_4a87_a2d9_9447e929538f.slice/crio-0522b95e8862e9cdc72101fdeba44c94ec8119b08d76c414a4936bc2856d28d8 WatchSource:0}: Error finding container 0522b95e8862e9cdc72101fdeba44c94ec8119b08d76c414a4936bc2856d28d8: Status 404 returned error can't find the container with id 0522b95e8862e9cdc72101fdeba44c94ec8119b08d76c414a4936bc2856d28d8 Nov 02 21:35:10.194092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.194065 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx"] Nov 02 21:35:10.195550 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:10.195517 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b13166_e7e0_4e41_ab46_ff31e5b49b29.slice/crio-b41c55ae6cfbd105a7842eefa3bf3465145fadad6c7521aa3ea49c9e33714c1a WatchSource:0}: Error finding container b41c55ae6cfbd105a7842eefa3bf3465145fadad6c7521aa3ea49c9e33714c1a: Status 404 returned error can't find the container with id b41c55ae6cfbd105a7842eefa3bf3465145fadad6c7521aa3ea49c9e33714c1a Nov 02 21:35:10.621634 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:35:10.621596 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:35:10.788067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.787521 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" event={"ID":"56fb272b-6df7-4a87-a2d9-9447e929538f","Type":"ContainerStarted","Data":"b8d2613684f85ba904713cd8e854aaa913b557cecc6f5d24860ee89381842cd8"} Nov 02 21:35:10.788067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.787583 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" event={"ID":"56fb272b-6df7-4a87-a2d9-9447e929538f","Type":"ContainerStarted","Data":"0522b95e8862e9cdc72101fdeba44c94ec8119b08d76c414a4936bc2856d28d8"} Nov 02 21:35:10.789945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.789914 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" event={"ID":"b9b13166-e7e0-4e41-ab46-ff31e5b49b29","Type":"ContainerStarted","Data":"74254bbba6181cdd981291816ee79555cef3a76e6171d0ee0300d3944ab77a61"} Nov 02 21:35:10.790102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.789988 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" event={"ID":"b9b13166-e7e0-4e41-ab46-ff31e5b49b29","Type":"ContainerStarted","Data":"b41c55ae6cfbd105a7842eefa3bf3465145fadad6c7521aa3ea49c9e33714c1a"} Nov 02 21:35:10.791865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.791838 4441 generic.go:356] "Generic (PLEG): container finished" podID="aaedf9a3-6eae-4435-9a5c-35a36ff16a40" containerID="75ddfbb745772c3c1498c1221d23b73b339e5e25a01a03ae3659edbd3656ad6d" exitCode=0 Nov 02 21:35:10.791995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.791926 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" event={"ID":"aaedf9a3-6eae-4435-9a5c-35a36ff16a40","Type":"ContainerDied","Data":"75ddfbb745772c3c1498c1221d23b73b339e5e25a01a03ae3659edbd3656ad6d"} Nov 02 21:35:10.834608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.834573 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m"] Nov 02 21:35:10.840249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.840224 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:10.909624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.898605 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m"] Nov 02 21:35:10.952875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.952829 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmx8w\" (UniqueName: \"kubernetes.io/projected/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-kube-api-access-rmx8w\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:10.953070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.953043 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:10.953203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:10.953176 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.033797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.033743 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" podStartSLOduration=2.033729195 podStartE2EDuration="2.033729195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:10.919686968 +0000 UTC m=+1340.475315307" watchObservedRunningTime="2026-11-02 21:35:11.033729195 +0000 UTC m=+1340.589357547" Nov 02 21:35:11.055091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.055000 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmx8w\" (UniqueName: \"kubernetes.io/projected/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-kube-api-access-rmx8w\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.055243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.055130 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.055243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.055194 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.056151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.056118 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.057714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.057687 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.109023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.108995 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmx8w\" (UniqueName: \"kubernetes.io/projected/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-kube-api-access-rmx8w\") pod \"maas-api-revoke-sub-eb6e3fd4-rgh9m\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.142716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.142669 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" podStartSLOduration=2.142655307 podStartE2EDuration="2.142655307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:11.13814076 +0000 UTC m=+1340.693769098" watchObservedRunningTime="2026-11-02 21:35:11.142655307 +0000 UTC m=+1340.698283643" Nov 02 21:35:11.155438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.155407 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:11.330218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.330175 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:11.331149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.331117 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:11.332107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.332087 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:35:11.489223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.489188 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:11.640665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.639475 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m"] Nov 02 21:35:11.803715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.803680 4441 generic.go:356] "Generic (PLEG): container finished" podID="56fb272b-6df7-4a87-a2d9-9447e929538f" containerID="b8d2613684f85ba904713cd8e854aaa913b557cecc6f5d24860ee89381842cd8" exitCode=0 Nov 02 21:35:11.803899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.803766 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" event={"ID":"56fb272b-6df7-4a87-a2d9-9447e929538f","Type":"ContainerDied","Data":"b8d2613684f85ba904713cd8e854aaa913b557cecc6f5d24860ee89381842cd8"} Nov 02 21:35:11.805398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.805368 4441 generic.go:356] "Generic (PLEG): container finished" podID="b9b13166-e7e0-4e41-ab46-ff31e5b49b29" containerID="74254bbba6181cdd981291816ee79555cef3a76e6171d0ee0300d3944ab77a61" exitCode=0 Nov 02 21:35:11.805534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.805450 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" event={"ID":"b9b13166-e7e0-4e41-ab46-ff31e5b49b29","Type":"ContainerDied","Data":"74254bbba6181cdd981291816ee79555cef3a76e6171d0ee0300d3944ab77a61"} Nov 02 21:35:11.806875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.806847 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" event={"ID":"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8","Type":"ContainerStarted","Data":"f9a3e17479991729bbf98db8ad68d89a3212d4e6a08e031d8f957800a1035646"} Nov 02 21:35:11.966946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:11.966920 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:12.089298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.089276 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-openshift-ca\") pod \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " Nov 02 21:35:12.089431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.089320 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-kube-api-access-zws8v\" (UniqueName: \"kubernetes.io/projected/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-kube-api-access-zws8v\") pod \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " Nov 02 21:35:12.089500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.089439 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-xks-ca\") pod \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\" (UID: \"aaedf9a3-6eae-4435-9a5c-35a36ff16a40\") " Nov 02 21:35:12.089678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.089647 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-openshift-ca" pod "aaedf9a3-6eae-4435-9a5c-35a36ff16a40" (UID: "aaedf9a3-6eae-4435-9a5c-35a36ff16a40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:12.090077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.090060 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:12.091292 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.091269 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-xks-ca" pod "aaedf9a3-6eae-4435-9a5c-35a36ff16a40" (UID: "aaedf9a3-6eae-4435-9a5c-35a36ff16a40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:12.091395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.091375 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-kube-api-access-zws8v" pod "aaedf9a3-6eae-4435-9a5c-35a36ff16a40" (UID: "aaedf9a3-6eae-4435-9a5c-35a36ff16a40"). InnerVolumeSpecName "kube-api-access-zws8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:12.191585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.191511 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zws8v\" (UniqueName: \"kubernetes.io/projected/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-kube-api-access-zws8v\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:12.191585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.191542 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaedf9a3-6eae-4435-9a5c-35a36ff16a40-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:12.843001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.838073 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" event={"ID":"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8","Type":"ContainerStarted","Data":"3f768160b881840d849021f61f633a7675947fdd204805de7de060c48383026a"} Nov 02 21:35:12.843001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.841440 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" Nov 02 21:35:12.857319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.852875 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c" event={"ID":"aaedf9a3-6eae-4435-9a5c-35a36ff16a40","Type":"ContainerDied","Data":"e02f0ac75032e93cec899c51600fbd5bd20beec5e85426ea984d55c7927e2a7a"} Nov 02 21:35:12.857319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.852939 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e02f0ac75032e93cec899c51600fbd5bd20beec5e85426ea984d55c7927e2a7a" Nov 02 21:35:12.959676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:12.959614 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" podStartSLOduration=2.959596119 podStartE2EDuration="2.959596119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:12.956249177 +0000 UTC m=+1342.511877537" watchObservedRunningTime="2026-11-02 21:35:12.959596119 +0000 UTC m=+1342.515224457" Nov 02 21:35:13.042023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.038802 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:13.214218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.214125 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:13.371405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.366243 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:13.511145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.510691 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:13.648441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.641783 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-xks-ca\") pod \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " Nov 02 21:35:13.648441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.641841 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-openshift-ca\") pod \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " Nov 02 21:35:13.663993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.655033 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-kube-api-access-4hv8g\" (UniqueName: \"kubernetes.io/projected/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-kube-api-access-4hv8g\") pod \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\" (UID: \"b9b13166-e7e0-4e41-ab46-ff31e5b49b29\") " Nov 02 21:35:13.663993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.657202 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-openshift-ca" pod "b9b13166-e7e0-4e41-ab46-ff31e5b49b29" (UID: "b9b13166-e7e0-4e41-ab46-ff31e5b49b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:13.714061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.708779 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:13.759776 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.759741 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:13.839698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.831511 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-xks-ca" pod "b9b13166-e7e0-4e41-ab46-ff31e5b49b29" (UID: "b9b13166-e7e0-4e41-ab46-ff31e5b49b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:13.892932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.892895 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:13.964108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:13.951797 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:14.145043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.130269 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-kube-api-access-4hv8g" pod "b9b13166-e7e0-4e41-ab46-ff31e5b49b29" (UID: "b9b13166-e7e0-4e41-ab46-ff31e5b49b29"). InnerVolumeSpecName "kube-api-access-4hv8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:14.214991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.214444 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hv8g\" (UniqueName: \"kubernetes.io/projected/b9b13166-e7e0-4e41-ab46-ff31e5b49b29-kube-api-access-4hv8g\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:14.296949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.271015 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:14.407092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.387500 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" event={"ID":"b9b13166-e7e0-4e41-ab46-ff31e5b49b29","Type":"ContainerDied","Data":"b41c55ae6cfbd105a7842eefa3bf3465145fadad6c7521aa3ea49c9e33714c1a"} Nov 02 21:35:14.407092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.387574 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b41c55ae6cfbd105a7842eefa3bf3465145fadad6c7521aa3ea49c9e33714c1a" Nov 02 21:35:14.407092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.387667 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx" Nov 02 21:35:14.407092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.393791 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.408380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.408344 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.434994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.415529 4441 generic.go:356] "Generic (PLEG): container finished" podID="cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" containerID="3f768160b881840d849021f61f633a7675947fdd204805de7de060c48383026a" exitCode=0 Nov 02 21:35:14.434994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.415734 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" event={"ID":"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8","Type":"ContainerDied","Data":"3f768160b881840d849021f61f633a7675947fdd204805de7de060c48383026a"} Nov 02 21:35:14.434994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.423047 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.458360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.447187 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.458755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.458719 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.502118 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.502079 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.508773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.504151 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:14.622756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.620625 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.660435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.660333 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.664092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.664053 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-openshift-ca\") pod \"56fb272b-6df7-4a87-a2d9-9447e929538f\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " Nov 02 21:35:14.664475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.664440 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fb272b-6df7-4a87-a2d9-9447e929538f-kube-api-access-tr49l\" (UniqueName: \"kubernetes.io/projected/56fb272b-6df7-4a87-a2d9-9447e929538f-kube-api-access-tr49l\") pod \"56fb272b-6df7-4a87-a2d9-9447e929538f\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " Nov 02 21:35:14.664717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.664694 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-xks-ca\") pod \"56fb272b-6df7-4a87-a2d9-9447e929538f\" (UID: \"56fb272b-6df7-4a87-a2d9-9447e929538f\") " Nov 02 21:35:14.665905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.665875 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:14.677801 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.669645 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Nov 02 21:35:14.712188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.703606 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-openshift-ca" pod "56fb272b-6df7-4a87-a2d9-9447e929538f" (UID: "56fb272b-6df7-4a87-a2d9-9447e929538f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:14.715991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.714733 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-xks-ca" pod "56fb272b-6df7-4a87-a2d9-9447e929538f" (UID: "56fb272b-6df7-4a87-a2d9-9447e929538f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:14.772258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.772158 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fb272b-6df7-4a87-a2d9-9447e929538f-kube-api-access-tr49l" pod "56fb272b-6df7-4a87-a2d9-9447e929538f" (UID: "56fb272b-6df7-4a87-a2d9-9447e929538f"). InnerVolumeSpecName "kube-api-access-tr49l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:14.775993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.775934 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr49l\" (UniqueName: \"kubernetes.io/projected/56fb272b-6df7-4a87-a2d9-9447e929538f-kube-api-access-tr49l\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:14.775993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.775977 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:14.775993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:14.775996 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb272b-6df7-4a87-a2d9-9447e929538f-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:15.358994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.335920 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 02 21:35:15.364011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.354308 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:15.393994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.368870 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.427993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.378494 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.490993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.459150 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.490993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.464048 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.498994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.495175 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.524692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.501986 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.832351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.524207 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.833144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.531807 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:35:15.896036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.872737 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:35:15.896036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.872892 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:35:15.896036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.873140 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:35:15.896036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.873205 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:35:15.953660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.692531 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Nov 02 21:35:15.953911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.534825 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 21:35:15.954077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.664609 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.954192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.870766 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.954315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.780333 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:35:15.977192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:15.805548 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.082219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.082167 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 02 21:35:16.181048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.165488 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:16.403487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.403445 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:35:16.434619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.434528 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.435709 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.435680 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:35:16.473901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.473855 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.476105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.476068 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.524429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.524395 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:35:16.550019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.542565 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:35:16.550019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.542629 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:35:16.550019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.542663 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:35:16.550019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.536415 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.738049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.699218 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.851870 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.851821 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.911994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.853643 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": context deadline exceeded" Nov 02 21:35:16.933034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.907378 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" Nov 02 21:35:16.933034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.907664 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:16.961346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.955408 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:35:16.961346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:16.955477 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.239995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.236037 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.295053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.261105 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.334992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.331717 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:17.480051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.426497 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:17.480051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.279153 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.480051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.261170 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 21:35:17.480051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.366010 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:35:17.539080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.484179 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.559069 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.563612 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.261191 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.600671 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.643073 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.643122 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj" event={"ID":"56fb272b-6df7-4a87-a2d9-9447e929538f","Type":"ContainerDied","Data":"0522b95e8862e9cdc72101fdeba44c94ec8119b08d76c414a4936bc2856d28d8"} Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.643165 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0522b95e8862e9cdc72101fdeba44c94ec8119b08d76c414a4936bc2856d28d8" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.643209 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.643229 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2"] Nov 02 21:35:17.647020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.643256 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e09d91-jhpt2"] Nov 02 21:35:17.900569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.900519 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.005044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.966223 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.005044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:17.993600 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 02 21:35:18.062702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.062660 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.276245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.276201 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-7bc68ffc7c-pmmp6" podUID="636400c6-9f4c-476d-964b-2bf5a279c586" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.49:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.301991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.301282 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.386955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.355617 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 21:35:18.386955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.366470 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.386955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.373012 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:18.431763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.429156 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.431763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.430567 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Nov 02 21:35:18.462049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.458690 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.487931 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.486480 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.531999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.514544 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.565020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.551071 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": context deadline exceeded" Nov 02 21:35:18.734083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.575818 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:18.734083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.605385 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.734083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.605519 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.734083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.605662 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.906279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.906229 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.116096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.871960 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423c1021-69e2-4d17-849a-290277942c76" path="/var/lib/kubelet/pods/423c1021-69e2-4d17-849a-290277942c76/volumes" Nov 02 21:35:19.116096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:18.968251 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.219991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.216053 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Nov 02 21:35:19.325125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.309259 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.382991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.379509 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:19.415697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.415656 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 21:35:19.438421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.433173 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.438832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.433219 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 21:35:19.500922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.482233 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:35:19.631266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.527937 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.918154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.918052 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:35:20.131166 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:35:19.583326 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fb272b_6df7_4a87_a2d9_9447e929538f.slice\": RecentStats: unable to find data in memory cache]" Nov 02 21:35:20.141475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.607051 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:35:20.156888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.037468 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:20.158073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.038213 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 02 21:35:20.158802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:19.579903 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:20.160188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.055350 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:20.470988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.470931 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Nov 02 21:35:20.486263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.486227 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 02 21:35:20.490552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.490514 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:20.509554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.509532 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:35:20.510067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.510050 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:35:20.510225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.510212 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:35:20.510387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.510376 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:35:20.510836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.510128 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:20.514046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.514026 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:35:20.514704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.514687 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" event={"ID":"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8","Type":"ContainerDied","Data":"f9a3e17479991729bbf98db8ad68d89a3212d4e6a08e031d8f957800a1035646"} Nov 02 21:35:20.516249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.516232 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a3e17479991729bbf98db8ad68d89a3212d4e6a08e031d8f957800a1035646" Nov 02 21:35:20.519611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.514612 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:20.521120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.521097 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:35:20.521262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.521153 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:35:20.523468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.523452 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:35:20.530484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.530469 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:20.537267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.537246 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:35:20.537977 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.537946 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:35:20.542168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.542149 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:35:20.543824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.543808 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:35:20.546471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.546453 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:35:20.561194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561173 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz"] Nov 02 21:35:20.561819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561799 4441 cpu_manager.go:414] "containerMap: removing container" podUID="aaedf9a3-6eae-4435-9a5c-35a36ff16a40" containerName="revoke-keys" Nov 02 21:35:20.561905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561820 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaedf9a3-6eae-4435-9a5c-35a36ff16a40" containerName="revoke-keys" Nov 02 21:35:20.561905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561835 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" containerName="revoke-keys" Nov 02 21:35:20.561905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561842 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" containerName="revoke-keys" Nov 02 21:35:20.561905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561871 4441 cpu_manager.go:414] "containerMap: removing container" podUID="56fb272b-6df7-4a87-a2d9-9447e929538f" containerName="revoke-keys" Nov 02 21:35:20.561905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561879 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fb272b-6df7-4a87-a2d9-9447e929538f" containerName="revoke-keys" Nov 02 21:35:20.561905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561897 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b9b13166-e7e0-4e41-ab46-ff31e5b49b29" containerName="revoke-keys" Nov 02 21:35:20.561905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.561905 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b13166-e7e0-4e41-ab46-ff31e5b49b29" containerName="revoke-keys" Nov 02 21:35:20.562322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.562156 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fb272b-6df7-4a87-a2d9-9447e929538f" containerName="revoke-keys" Nov 02 21:35:20.562322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.562173 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b13166-e7e0-4e41-ab46-ff31e5b49b29" containerName="revoke-keys" Nov 02 21:35:20.562322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.562193 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaedf9a3-6eae-4435-9a5c-35a36ff16a40" containerName="revoke-keys" Nov 02 21:35:20.562322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.562204 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" containerName="revoke-keys" Nov 02 21:35:20.565842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.565812 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.566150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.566129 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz"] Nov 02 21:35:20.670794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.670764 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-xks-ca\") pod \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " Nov 02 21:35:20.671009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.670834 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-openshift-ca\") pod \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " Nov 02 21:35:20.671103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.671023 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-kube-api-access-rmx8w\" (UniqueName: \"kubernetes.io/projected/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-kube-api-access-rmx8w\") pod \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\" (UID: \"cfdf6f47-fe9c-4676-90f1-9da71e75c1d8\") " Nov 02 21:35:20.671311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.671279 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-openshift-ca" pod "cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" (UID: "cfdf6f47-fe9c-4676-90f1-9da71e75c1d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:20.673074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.671600 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.673074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.671659 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.673074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.671724 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7ds\" (UniqueName: \"kubernetes.io/projected/4cc53ef2-3cf2-4417-a110-d76abc393828-kube-api-access-gf7ds\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.673956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.673837 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.676000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.675961 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-kube-api-access-rmx8w" pod "cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" (UID: "cfdf6f47-fe9c-4676-90f1-9da71e75c1d8"). InnerVolumeSpecName "kube-api-access-rmx8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.676401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.676386 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-xks-ca" pod "cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" (UID: "cfdf6f47-fe9c-4676-90f1-9da71e75c1d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:20.775947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.775849 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.775947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.775902 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.775947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.775942 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7ds\" (UniqueName: \"kubernetes.io/projected/4cc53ef2-3cf2-4417-a110-d76abc393828-kube-api-access-gf7ds\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.776270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.776203 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.776270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.776221 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmx8w\" (UniqueName: \"kubernetes.io/projected/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8-kube-api-access-rmx8w\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.776740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.776709 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.778860 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.778831 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.829534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.829504 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7ds\" (UniqueName: \"kubernetes.io/projected/4cc53ef2-3cf2-4417-a110-d76abc393828-kube-api-access-gf7ds\") pod \"maas-api-revoke-sub-84eb51c4-c5tjz\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:20.886072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:20.886016 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:21.300050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:21.300019 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz"] Nov 02 21:35:21.301621 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:21.301585 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc53ef2_3cf2_4417_a110_d76abc393828.slice/crio-ff8615e259ef089158a8255b2856727ac7289e6fc7cb140ced9c537087e4350e WatchSource:0}: Error finding container ff8615e259ef089158a8255b2856727ac7289e6fc7cb140ced9c537087e4350e: Status 404 returned error can't find the container with id ff8615e259ef089158a8255b2856727ac7289e6fc7cb140ced9c537087e4350e Nov 02 21:35:21.546176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:21.546141 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" event={"ID":"4cc53ef2-3cf2-4417-a110-d76abc393828","Type":"ContainerStarted","Data":"ff8615e259ef089158a8255b2856727ac7289e6fc7cb140ced9c537087e4350e"} Nov 02 21:35:21.546694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:21.546446 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m" Nov 02 21:35:22.278989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:22.278945 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:22.485780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:22.485712 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:22.556355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:22.556270 4441 generic.go:356] "Generic (PLEG): container finished" podID="4cc53ef2-3cf2-4417-a110-d76abc393828" containerID="31390cf4a1d9738b4a2560d140acb1f5d2dac8fe3fda2d5829ef7187c7b0a94b" exitCode=0 Nov 02 21:35:22.556355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:22.556336 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" event={"ID":"4cc53ef2-3cf2-4417-a110-d76abc393828","Type":"ContainerDied","Data":"31390cf4a1d9738b4a2560d140acb1f5d2dac8fe3fda2d5829ef7187c7b0a94b"} Nov 02 21:35:22.613838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:22.613803 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.196355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.196325 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.345181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.345149 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.616296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.613166 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.742091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.742067 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:23.768910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.768856 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.828600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.828543 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-openshift-ca\") pod \"4cc53ef2-3cf2-4417-a110-d76abc393828\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " Nov 02 21:35:23.828716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.828622 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-xks-ca\") pod \"4cc53ef2-3cf2-4417-a110-d76abc393828\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " Nov 02 21:35:23.828766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.828744 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc53ef2-3cf2-4417-a110-d76abc393828-kube-api-access-gf7ds\" (UniqueName: \"kubernetes.io/projected/4cc53ef2-3cf2-4417-a110-d76abc393828-kube-api-access-gf7ds\") pod \"4cc53ef2-3cf2-4417-a110-d76abc393828\" (UID: \"4cc53ef2-3cf2-4417-a110-d76abc393828\") " Nov 02 21:35:23.828935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.828914 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-openshift-ca" pod "4cc53ef2-3cf2-4417-a110-d76abc393828" (UID: "4cc53ef2-3cf2-4417-a110-d76abc393828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:23.829410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.829388 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:23.830708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.830684 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc53ef2-3cf2-4417-a110-d76abc393828-kube-api-access-gf7ds" pod "4cc53ef2-3cf2-4417-a110-d76abc393828" (UID: "4cc53ef2-3cf2-4417-a110-d76abc393828"). InnerVolumeSpecName "kube-api-access-gf7ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:23.830708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.830695 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-xks-ca" pod "4cc53ef2-3cf2-4417-a110-d76abc393828" (UID: "4cc53ef2-3cf2-4417-a110-d76abc393828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:23.912752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.912716 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.930635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.930610 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc53ef2-3cf2-4417-a110-d76abc393828-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:23.930635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:23.930635 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7ds\" (UniqueName: \"kubernetes.io/projected/4cc53ef2-3cf2-4417-a110-d76abc393828-kube-api-access-gf7ds\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:24.145525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.145450 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh"] Nov 02 21:35:24.146103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.146088 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4cc53ef2-3cf2-4417-a110-d76abc393828" containerName="revoke-keys" Nov 02 21:35:24.146163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.146105 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc53ef2-3cf2-4417-a110-d76abc393828" containerName="revoke-keys" Nov 02 21:35:24.146234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.146225 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc53ef2-3cf2-4417-a110-d76abc393828" containerName="revoke-keys" Nov 02 21:35:24.150122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.150092 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.315243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.315208 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4dff0429-openshift-default-dockercfg-s5pmw\"" Nov 02 21:35:24.388675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.388616 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-socket\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.394212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.394135 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-credential-socket\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.394423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.394404 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-envoy\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.394634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.394616 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-certs\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.394788 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.394768 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5hg\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-kube-api-access-gf5hg\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.395031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.395012 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-data\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.395192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.395173 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-token\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.395317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.395301 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-podinfo\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.395471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.395453 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc184427-21af-49cc-b2fd-07f5426aedaf-istiod-ca-cert\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.435906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.435817 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh"] Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498298 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-socket\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498401 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-credential-socket\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498434 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-envoy\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498518 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-certs\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498561 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5hg\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-kube-api-access-gf5hg\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498671 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-data\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498700 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-token\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498728 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-podinfo\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.498782 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc184427-21af-49cc-b2fd-07f5426aedaf-istiod-ca-cert\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.499753 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc184427-21af-49cc-b2fd-07f5426aedaf-istiod-ca-cert\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.500055 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-socket\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.500623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.500280 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-credential-socket\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.502009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.501957 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-data\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.515280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.502386 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-certs\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.528598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.517071 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-podinfo\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.531543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.529179 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-envoy\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.556408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.556372 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-token\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.556579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.556500 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5hg\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-kube-api-access-gf5hg\") pod \"e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:24.589022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.588486 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" Nov 02 21:35:24.591015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.590943 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz" event={"ID":"4cc53ef2-3cf2-4417-a110-d76abc393828","Type":"ContainerDied","Data":"ff8615e259ef089158a8255b2856727ac7289e6fc7cb140ced9c537087e4350e"} Nov 02 21:35:24.591015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.591018 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8615e259ef089158a8255b2856727ac7289e6fc7cb140ced9c537087e4350e" Nov 02 21:35:24.764042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:24.763928 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:25.208273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.208236 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh"] Nov 02 21:35:25.208538 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:25.208503 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc184427_21af_49cc_b2fd_07f5426aedaf.slice/crio-6b89efe2d2002274745f96836f5c659767c045f3a63095dbca775fa1362df58c WatchSource:0}: Error finding container 6b89efe2d2002274745f96836f5c659767c045f3a63095dbca775fa1362df58c: Status 404 returned error can't find the container with id 6b89efe2d2002274745f96836f5c659767c045f3a63095dbca775fa1362df58c Nov 02 21:35:25.211471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.211431 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:25.211557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.211518 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:25.211596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.211558 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:25.606272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.606227 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" event={"ID":"dc184427-21af-49cc-b2fd-07f5426aedaf","Type":"ContainerStarted","Data":"99acbefa13ac9e1aa413d2dfa618f790376b09692a9dd4d806be42af73850761"} Nov 02 21:35:25.606455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.606298 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" event={"ID":"dc184427-21af-49cc-b2fd-07f5426aedaf","Type":"ContainerStarted","Data":"6b89efe2d2002274745f96836f5c659767c045f3a63095dbca775fa1362df58c"} Nov 02 21:35:25.617877 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.617847 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:25.721914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.721860 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podStartSLOduration=1.721845376 podStartE2EDuration="1.721845376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:25.716225296 +0000 UTC m=+1355.271853640" watchObservedRunningTime="2026-11-02 21:35:25.721845376 +0000 UTC m=+1355.277473712" Nov 02 21:35:25.747292 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.747251 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:25.764063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:25.764032 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:26.770061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:26.770033 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:27.311622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.311592 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:27.554994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.553629 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:27.635326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.635231 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:27.636258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.636234 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:35:27.768652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.768616 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv"] Nov 02 21:35:27.773906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.773883 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:27.776416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.776389 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x"] Nov 02 21:35:27.788102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.788077 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:27.798046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.797248 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv"] Nov 02 21:35:27.826032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.825306 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2ccb9fb8\"" Nov 02 21:35:27.826032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.825464 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2ccb9fb8-dockercfg-stdwm\"" Nov 02 21:35:27.829384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.829359 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x"] Nov 02 21:35:27.849188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.849153 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:35:27.849374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.849249 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:35:27.849436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.849394 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:35:27.849486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.849459 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:35:27.865218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.864435 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgdx\" (UniqueName: \"kubernetes.io/projected/facab52c-2e15-4b8e-bdba-99df0b3703c4-kube-api-access-5dgdx\") pod \"payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:27.865218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.864751 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:27.865218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.864992 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:27.865218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.865065 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facab52c-2e15-4b8e-bdba-99df0b3703c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:27.865218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:27.865110 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7x8q\" (UniqueName: \"kubernetes.io/projected/d25b3dff-2d67-4683-9d59-5b994ed91792-kube-api-access-q7x8q\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.096844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.094054 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgdx\" (UniqueName: \"kubernetes.io/projected/facab52c-2e15-4b8e-bdba-99df0b3703c4-kube-api-access-5dgdx\") pod \"payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:28.096844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.094179 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.096844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.094330 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.096844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.094498 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facab52c-2e15-4b8e-bdba-99df0b3703c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:28.096844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.094544 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7x8q\" (UniqueName: \"kubernetes.io/projected/d25b3dff-2d67-4683-9d59-5b994ed91792-kube-api-access-q7x8q\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.096844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.095737 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facab52c-2e15-4b8e-bdba-99df0b3703c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:28.096844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.096526 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.107036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.103622 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.149223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.149190 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7x8q\" (UniqueName: \"kubernetes.io/projected/d25b3dff-2d67-4683-9d59-5b994ed91792-kube-api-access-q7x8q\") pod \"maas-api-revoke-sub-2eca6925-zgm6x\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.149377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.149190 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgdx\" (UniqueName: \"kubernetes.io/projected/facab52c-2e15-4b8e-bdba-99df0b3703c4-kube-api-access-5dgdx\") pod \"payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:28.404090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.404022 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:28.427939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.427914 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:28.794763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.794692 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv"] Nov 02 21:35:28.841029 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:28.840951 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacab52c_2e15_4b8e_bdba_99df0b3703c4.slice/crio-54440c32d0aa101eae2247e72267c4c9171fd2e0617d08083f5a01619d598da8 WatchSource:0}: Error finding container 54440c32d0aa101eae2247e72267c4c9171fd2e0617d08083f5a01619d598da8: Status 404 returned error can't find the container with id 54440c32d0aa101eae2247e72267c4c9171fd2e0617d08083f5a01619d598da8 Nov 02 21:35:28.849027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:28.848999 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x"] Nov 02 21:35:28.861547 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:28.861518 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25b3dff_2d67_4683_9d59_5b994ed91792.slice/crio-9daeab7159c8fc702de87372b0c77449528b0c916b90f00159301373f9c35f56 WatchSource:0}: Error finding container 9daeab7159c8fc702de87372b0c77449528b0c916b90f00159301373f9c35f56: Status 404 returned error can't find the container with id 9daeab7159c8fc702de87372b0c77449528b0c916b90f00159301373f9c35f56 Nov 02 21:35:29.217308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.217276 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:29.407675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.407640 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:29.530537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.530462 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:29.659759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.659720 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" event={"ID":"facab52c-2e15-4b8e-bdba-99df0b3703c4","Type":"ContainerStarted","Data":"c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f"} Nov 02 21:35:29.659955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.659812 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" event={"ID":"facab52c-2e15-4b8e-bdba-99df0b3703c4","Type":"ContainerStarted","Data":"54440c32d0aa101eae2247e72267c4c9171fd2e0617d08083f5a01619d598da8"} Nov 02 21:35:29.660088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.660027 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:29.662174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.662142 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" event={"ID":"d25b3dff-2d67-4683-9d59-5b994ed91792","Type":"ContainerStarted","Data":"1e727870e6a321b307bda108737f84110666c4a777ba22453ca16891215d4878"} Nov 02 21:35:29.662290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.662198 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" event={"ID":"d25b3dff-2d67-4683-9d59-5b994ed91792","Type":"ContainerStarted","Data":"9daeab7159c8fc702de87372b0c77449528b0c916b90f00159301373f9c35f56"} Nov 02 21:35:29.662673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.662651 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:29.785053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.784934 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" podStartSLOduration=2.784918383 podStartE2EDuration="2.784918383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:29.782843472 +0000 UTC m=+1359.338471809" watchObservedRunningTime="2026-11-02 21:35:29.784918383 +0000 UTC m=+1359.340546756" Nov 02 21:35:29.897337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:29.897284 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" podStartSLOduration=2.8972700209999998 podStartE2EDuration="2.897270021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:29.893358802 +0000 UTC m=+1359.448987140" watchObservedRunningTime="2026-11-02 21:35:29.897270021 +0000 UTC m=+1359.452898357" Nov 02 21:35:30.674957 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:30.674922 4441 generic.go:356] "Generic (PLEG): container finished" podID="d25b3dff-2d67-4683-9d59-5b994ed91792" containerID="1e727870e6a321b307bda108737f84110666c4a777ba22453ca16891215d4878" exitCode=0 Nov 02 21:35:30.675186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:30.675002 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" event={"ID":"d25b3dff-2d67-4683-9d59-5b994ed91792","Type":"ContainerDied","Data":"1e727870e6a321b307bda108737f84110666c4a777ba22453ca16891215d4878"} Nov 02 21:35:31.851176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.851151 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:31.964779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.964530 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-openshift-ca\") pod \"d25b3dff-2d67-4683-9d59-5b994ed91792\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " Nov 02 21:35:31.965499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.965454 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25b3dff-2d67-4683-9d59-5b994ed91792-kube-api-access-q7x8q\" (UniqueName: \"kubernetes.io/projected/d25b3dff-2d67-4683-9d59-5b994ed91792-kube-api-access-q7x8q\") pod \"d25b3dff-2d67-4683-9d59-5b994ed91792\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " Nov 02 21:35:31.965793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.965768 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-xks-ca\") pod \"d25b3dff-2d67-4683-9d59-5b994ed91792\" (UID: \"d25b3dff-2d67-4683-9d59-5b994ed91792\") " Nov 02 21:35:31.967733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.967709 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-openshift-ca" pod "d25b3dff-2d67-4683-9d59-5b994ed91792" (UID: "d25b3dff-2d67-4683-9d59-5b994ed91792"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:31.969536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.969514 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-xks-ca" pod "d25b3dff-2d67-4683-9d59-5b994ed91792" (UID: "d25b3dff-2d67-4683-9d59-5b994ed91792"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:31.970205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.970185 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25b3dff-2d67-4683-9d59-5b994ed91792-kube-api-access-q7x8q" pod "d25b3dff-2d67-4683-9d59-5b994ed91792" (UID: "d25b3dff-2d67-4683-9d59-5b994ed91792"). InnerVolumeSpecName "kube-api-access-q7x8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.987757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:31.987731 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:32.068920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.068889 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:32.068920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.068918 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25b3dff-2d67-4683-9d59-5b994ed91792-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:32.069173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.068930 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7x8q\" (UniqueName: \"kubernetes.io/projected/d25b3dff-2d67-4683-9d59-5b994ed91792-kube-api-access-q7x8q\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:32.178990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.176526 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:32.424988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.424040 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f"] Nov 02 21:35:32.428989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.425218 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d25b3dff-2d67-4683-9d59-5b994ed91792" containerName="revoke-keys" Nov 02 21:35:32.428989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.425243 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25b3dff-2d67-4683-9d59-5b994ed91792" containerName="revoke-keys" Nov 02 21:35:32.428989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.425406 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25b3dff-2d67-4683-9d59-5b994ed91792" containerName="revoke-keys" Nov 02 21:35:32.452989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.446891 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.476993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.469899 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f"] Nov 02 21:35:32.493112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.493037 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820029e7-2538-44c4-bd43-a3ad4bae01b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.493285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.493253 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4kv\" (UniqueName: \"kubernetes.io/projected/820029e7-2538-44c4-bd43-a3ad4bae01b3-kube-api-access-dd4kv\") pod \"payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.506586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.506561 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c39c4123-dockercfg-hsrtj\"" Nov 02 21:35:32.508138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.508102 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c39c4123\"" Nov 02 21:35:32.562879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.560363 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55"] Nov 02 21:35:32.591078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.581857 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:32.595545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.595505 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820029e7-2538-44c4-bd43-a3ad4bae01b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.595711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.595674 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4kv\" (UniqueName: \"kubernetes.io/projected/820029e7-2538-44c4-bd43-a3ad4bae01b3-kube-api-access-dd4kv\") pod \"payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.600001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.599948 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820029e7-2538-44c4-bd43-a3ad4bae01b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.719706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.696304 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4kv\" (UniqueName: \"kubernetes.io/projected/820029e7-2538-44c4-bd43-a3ad4bae01b3-kube-api-access-dd4kv\") pod \"payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.719706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.703144 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3865cb-8635-4023-bbe9-632c2095371a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:32.719706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.703206 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzhs\" (UniqueName: \"kubernetes.io/projected/6a3865cb-8635-4023-bbe9-632c2095371a-kube-api-access-bwzhs\") pod \"payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:32.719706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.718750 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55"] Nov 02 21:35:32.802751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.802717 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:32.806231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.806196 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3865cb-8635-4023-bbe9-632c2095371a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:32.806534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.806513 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzhs\" (UniqueName: \"kubernetes.io/projected/6a3865cb-8635-4023-bbe9-632c2095371a-kube-api-access-bwzhs\") pod \"payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:32.807782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.807581 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3865cb-8635-4023-bbe9-632c2095371a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:32.829467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.828278 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" event={"ID":"d25b3dff-2d67-4683-9d59-5b994ed91792","Type":"ContainerDied","Data":"9daeab7159c8fc702de87372b0c77449528b0c916b90f00159301373f9c35f56"} Nov 02 21:35:32.829467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.828344 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9daeab7159c8fc702de87372b0c77449528b0c916b90f00159301373f9c35f56" Nov 02 21:35:32.829467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.828442 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x" Nov 02 21:35:32.861637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:32.861600 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzhs\" (UniqueName: \"kubernetes.io/projected/6a3865cb-8635-4023-bbe9-632c2095371a-kube-api-access-bwzhs\") pod \"payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:33.004578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.004494 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:33.236106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.236076 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f"] Nov 02 21:35:33.239682 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:33.239648 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820029e7_2538_44c4_bd43_a3ad4bae01b3.slice/crio-16cca3030cb9f6001c8010530e609f470ee265256df5652b58398876dcb953a9 WatchSource:0}: Error finding container 16cca3030cb9f6001c8010530e609f470ee265256df5652b58398876dcb953a9: Status 404 returned error can't find the container with id 16cca3030cb9f6001c8010530e609f470ee265256df5652b58398876dcb953a9 Nov 02 21:35:33.394620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.394522 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55"] Nov 02 21:35:33.397216 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:33.397188 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3865cb_8635_4023_bbe9_632c2095371a.slice/crio-080d434d39f9c0d5dc354a9838453e65089ad7aee6ff97c451a619780283b551 WatchSource:0}: Error finding container 080d434d39f9c0d5dc354a9838453e65089ad7aee6ff97c451a619780283b551: Status 404 returned error can't find the container with id 080d434d39f9c0d5dc354a9838453e65089ad7aee6ff97c451a619780283b551 Nov 02 21:35:33.852844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.852802 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" event={"ID":"6a3865cb-8635-4023-bbe9-632c2095371a","Type":"ContainerStarted","Data":"30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81"} Nov 02 21:35:33.853182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.853149 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" event={"ID":"6a3865cb-8635-4023-bbe9-632c2095371a","Type":"ContainerStarted","Data":"080d434d39f9c0d5dc354a9838453e65089ad7aee6ff97c451a619780283b551"} Nov 02 21:35:33.853313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.853227 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:33.855774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.855743 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" event={"ID":"820029e7-2538-44c4-bd43-a3ad4bae01b3","Type":"ContainerStarted","Data":"9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a"} Nov 02 21:35:33.855897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.855796 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" event={"ID":"820029e7-2538-44c4-bd43-a3ad4bae01b3","Type":"ContainerStarted","Data":"16cca3030cb9f6001c8010530e609f470ee265256df5652b58398876dcb953a9"} Nov 02 21:35:33.855897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.855828 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:33.907334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.907300 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:33.969584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:33.969538 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" podStartSLOduration=1.969525473 podStartE2EDuration="1.969525473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:33.966552103 +0000 UTC m=+1363.522180440" watchObservedRunningTime="2026-11-02 21:35:33.969525473 +0000 UTC m=+1363.525153809" Nov 02 21:35:34.040711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.040681 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:34.079959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.079923 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz"] Nov 02 21:35:34.083563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.083517 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" podStartSLOduration=2.083504962 podStartE2EDuration="2.083504962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:34.075662389 +0000 UTC m=+1363.631290728" watchObservedRunningTime="2026-11-02 21:35:34.083504962 +0000 UTC m=+1363.639133323" Nov 02 21:35:34.085758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.085738 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.204829 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.204913 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.204956 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.205002 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5p8\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-kube-api-access-mg5p8\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.205027 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.205074 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.205138 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.205201 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.213046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.205233 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.238402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.227143 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz"] Nov 02 21:35:34.238402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.236891 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:34.319264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.319227 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.319729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.319702 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.320440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.320409 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.321162 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.321144 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5p8\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-kube-api-access-mg5p8\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.322235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.322162 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.324608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.322396 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.324874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.322795 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.324874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.320334 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.324874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.321080 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.325358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.325265 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.326164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.326146 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.326681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.326663 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.326873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.326853 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.327632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.327572 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.328329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.328312 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.345063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.345007 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.380006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.375945 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5p8\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-kube-api-access-mg5p8\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.380168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.380153 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.423870 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.423839 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:34.881290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.881252 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz"] Nov 02 21:35:34.884562 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:34.884530 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45ffb6e_c4bc_495a_b50d_42c41d6842fe.slice/crio-94285668477a0a149d065a21032cb725ab78b4882017bf4a23f6c4dee4ea3b37 WatchSource:0}: Error finding container 94285668477a0a149d065a21032cb725ab78b4882017bf4a23f6c4dee4ea3b37: Status 404 returned error can't find the container with id 94285668477a0a149d065a21032cb725ab78b4882017bf4a23f6c4dee4ea3b37 Nov 02 21:35:34.887326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.887293 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:34.887429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.887357 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:34.887429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.887386 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:34.915689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:34.915657 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:35.215507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:35.215426 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:35.350447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:35.350413 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:35.477163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:35.477041 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:35.658387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:35.658342 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:35.885791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:35.885750 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" event={"ID":"d45ffb6e-c4bc-495a-b50d-42c41d6842fe","Type":"ContainerStarted","Data":"d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1"} Nov 02 21:35:35.888297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:35.888261 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" event={"ID":"d45ffb6e-c4bc-495a-b50d-42c41d6842fe","Type":"ContainerStarted","Data":"94285668477a0a149d065a21032cb725ab78b4882017bf4a23f6c4dee4ea3b37"} Nov 02 21:35:36.103064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.103007 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podStartSLOduration=2.102958326 podStartE2EDuration="2.102958326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:36.053251442 +0000 UTC m=+1365.608879780" watchObservedRunningTime="2026-11-02 21:35:36.102958326 +0000 UTC m=+1365.658586663" Nov 02 21:35:36.115078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.115047 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm"] Nov 02 21:35:36.119777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.119754 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.162087 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-certs\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.166093 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-envoy\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.166283 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a8a72e-7779-4051-ad2e-5b1669788505-istio-podinfo\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.166847 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxws\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-kube-api-access-pnxws\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.166993 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a8a72e-7779-4051-ad2e-5b1669788505-istiod-ca-cert\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.167026 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-data\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.167085 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-credential-socket\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.167110 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-istio-token\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.169047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.168544 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-socket\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.202651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.184150 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c7cd2d37-openshift-default-dockercfg-44ws6\"" Nov 02 21:35:36.269015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.256466 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.269673 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-envoy\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.269772 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a8a72e-7779-4051-ad2e-5b1669788505-istio-podinfo\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.269814 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxws\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-kube-api-access-pnxws\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.269896 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a8a72e-7779-4051-ad2e-5b1669788505-istiod-ca-cert\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.269923 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-data\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.270019 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-credential-socket\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.270048 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-istio-token\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.270188 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-socket\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.270339 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-certs\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.297375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.270801 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-certs\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.318024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.313036 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm"] Nov 02 21:35:36.318024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.313487 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-data\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.336906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.336634 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-credential-socket\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.336906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.336874 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-socket\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.347998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.344564 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a8a72e-7779-4051-ad2e-5b1669788505-istio-podinfo\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.366003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.355272 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-envoy\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.366003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.358668 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a8a72e-7779-4051-ad2e-5b1669788505-istiod-ca-cert\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.429050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.424662 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:36.430575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.430107 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 21:35:36.430575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.430162 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 21:35:36.433987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.431165 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-istio-token\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.439643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.439613 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxws\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-kube-api-access-pnxws\") pod \"e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.737186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.737091 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:36.846960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:36.846914 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:37.032131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.032008 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:37.235381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.235343 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm"] Nov 02 21:35:37.249296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.249261 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:37.249423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.249344 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:37.249423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.249388 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:37.430690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.430642 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m"] Nov 02 21:35:37.431523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.431488 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 21:35:37.431659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.431548 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 21:35:37.449263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.449232 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.523179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.523037 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m"] Nov 02 21:35:37.538026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.537982 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4dff0429\"" Nov 02 21:35:37.538431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.538376 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4dff0429-dockercfg-8cgtn\"" Nov 02 21:35:37.562009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.556995 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbk57\" (UniqueName: \"kubernetes.io/projected/8c009ac0-b1cc-48ec-ba41-9792bd563285-kube-api-access-xbk57\") pod \"payload-processing-e2e-derive-4dff0429-cc8778797-59x9m\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.562009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.557047 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c009ac0-b1cc-48ec-ba41-9792bd563285-plugins-config-volume\") pod \"payload-processing-e2e-derive-4dff0429-cc8778797-59x9m\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.658888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.658849 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbk57\" (UniqueName: \"kubernetes.io/projected/8c009ac0-b1cc-48ec-ba41-9792bd563285-kube-api-access-xbk57\") pod \"payload-processing-e2e-derive-4dff0429-cc8778797-59x9m\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.658888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.658892 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c009ac0-b1cc-48ec-ba41-9792bd563285-plugins-config-volume\") pod \"payload-processing-e2e-derive-4dff0429-cc8778797-59x9m\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.659784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.659758 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c009ac0-b1cc-48ec-ba41-9792bd563285-plugins-config-volume\") pod \"payload-processing-e2e-derive-4dff0429-cc8778797-59x9m\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.726916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.726882 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbk57\" (UniqueName: \"kubernetes.io/projected/8c009ac0-b1cc-48ec-ba41-9792bd563285-kube-api-access-xbk57\") pod \"payload-processing-e2e-derive-4dff0429-cc8778797-59x9m\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.915991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.915335 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:37.930987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:37.929463 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:38.056846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.056805 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" event={"ID":"85a8a72e-7779-4051-ad2e-5b1669788505","Type":"ContainerStarted","Data":"c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7"} Nov 02 21:35:38.057159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.057137 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" event={"ID":"85a8a72e-7779-4051-ad2e-5b1669788505","Type":"ContainerStarted","Data":"f3753253c23c90165ef4ce669ab259c325e6793f4ed36c2514d3347618f83bd8"} Nov 02 21:35:38.058946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.058921 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8"] Nov 02 21:35:38.071606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.071229 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.101182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.101154 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8"] Nov 02 21:35:38.134093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.126349 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:35:38.134093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.126505 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:35:38.134093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.126632 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:35:38.134093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.126727 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:35:38.181410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.177463 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" podStartSLOduration=2.177442226 podStartE2EDuration="2.177442226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:38.167049615 +0000 UTC m=+1367.722677953" watchObservedRunningTime="2026-11-02 21:35:38.177442226 +0000 UTC m=+1367.733070564" Nov 02 21:35:38.264829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.264780 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlrn\" (UniqueName: \"kubernetes.io/projected/252fabe8-441b-42b5-ad3d-0ad9c29815e6-kube-api-access-szlrn\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.269298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.267088 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.269298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.267144 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.390999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.383629 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlrn\" (UniqueName: \"kubernetes.io/projected/252fabe8-441b-42b5-ad3d-0ad9c29815e6-kube-api-access-szlrn\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.390999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.383746 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.390999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.383788 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.430149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.392510 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.432248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.432150 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 21:35:38.568145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.568099 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 21:35:38.568403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.457775 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.568546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.483676 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlrn\" (UniqueName: \"kubernetes.io/projected/252fabe8-441b-42b5-ad3d-0ad9c29815e6-kube-api-access-szlrn\") pod \"maas-api-revoke-sub-0fd8d12a-mcvd8\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.704036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.703318 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:38.738454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.737778 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:38.753896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.752228 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:38.868290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:38.867017 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m"] Nov 02 21:35:39.119789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.119486 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" event={"ID":"8c009ac0-b1cc-48ec-ba41-9792bd563285","Type":"ContainerStarted","Data":"3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf"} Nov 02 21:35:39.120258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.119558 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" event={"ID":"8c009ac0-b1cc-48ec-ba41-9792bd563285","Type":"ContainerStarted","Data":"c18d7a59d27f2b8561afeb95c0b9574db90bef16faa7e10cbe564246b3f42666"} Nov 02 21:35:39.120258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.120050 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:39.120258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.120119 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:39.120258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.120226 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:35:39.168803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.168775 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8"] Nov 02 21:35:39.213062 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:39.211256 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252fabe8_441b_42b5_ad3d_0ad9c29815e6.slice/crio-5af02467b27242fd960338426bc2fbd17c2c07ef6199e86f26883d519d79e25f WatchSource:0}: Error finding container 5af02467b27242fd960338426bc2fbd17c2c07ef6199e86f26883d519d79e25f: Status 404 returned error can't find the container with id 5af02467b27242fd960338426bc2fbd17c2c07ef6199e86f26883d519d79e25f Nov 02 21:35:39.256347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.256298 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" podStartSLOduration=2.256284081 podStartE2EDuration="2.256284081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:39.25407883 +0000 UTC m=+1368.809707167" watchObservedRunningTime="2026-11-02 21:35:39.256284081 +0000 UTC m=+1368.811912418" Nov 02 21:35:39.424711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.424574 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 21:35:39.424711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:39.424629 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 21:35:40.131209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.131178 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" event={"ID":"252fabe8-441b-42b5-ad3d-0ad9c29815e6","Type":"ContainerStarted","Data":"f734db3ea10afe1d592faa65e0c1a79a2c556910f7ca336678e7cef5f143b6ea"} Nov 02 21:35:40.131390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.131241 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" event={"ID":"252fabe8-441b-42b5-ad3d-0ad9c29815e6","Type":"ContainerStarted","Data":"5af02467b27242fd960338426bc2fbd17c2c07ef6199e86f26883d519d79e25f"} Nov 02 21:35:40.245335 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.245282 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" podStartSLOduration=3.245267125 podStartE2EDuration="3.245267125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:40.242753727 +0000 UTC m=+1369.798382076" watchObservedRunningTime="2026-11-02 21:35:40.245267125 +0000 UTC m=+1369.800895462" Nov 02 21:35:40.424309 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.424275 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 21:35:40.424416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.424335 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 21:35:40.432257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.432235 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:40.608174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.608026 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:40.692346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:40.692261 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:35:40.932632 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:35:40.932595 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:35:41.117836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.117800 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:41.144156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.144121 4441 generic.go:356] "Generic (PLEG): container finished" podID="252fabe8-441b-42b5-ad3d-0ad9c29815e6" containerID="f734db3ea10afe1d592faa65e0c1a79a2c556910f7ca336678e7cef5f143b6ea" exitCode=0 Nov 02 21:35:41.144323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.144219 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" event={"ID":"252fabe8-441b-42b5-ad3d-0ad9c29815e6","Type":"ContainerDied","Data":"f734db3ea10afe1d592faa65e0c1a79a2c556910f7ca336678e7cef5f143b6ea"} Nov 02 21:35:41.250989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.250897 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:41.425097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.425016 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 21:35:41.425243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.425095 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 21:35:41.446003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.440356 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:41.570022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:41.569988 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:42.371242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.371214 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:42.444916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.444836 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252fabe8-441b-42b5-ad3d-0ad9c29815e6-kube-api-access-szlrn\" (UniqueName: \"kubernetes.io/projected/252fabe8-441b-42b5-ad3d-0ad9c29815e6-kube-api-access-szlrn\") pod \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " Nov 02 21:35:42.445102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.444956 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-xks-ca\") pod \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " Nov 02 21:35:42.445102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.445022 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-openshift-ca\") pod \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\" (UID: \"252fabe8-441b-42b5-ad3d-0ad9c29815e6\") " Nov 02 21:35:42.445437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.445408 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-openshift-ca" pod "252fabe8-441b-42b5-ad3d-0ad9c29815e6" (UID: "252fabe8-441b-42b5-ad3d-0ad9c29815e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:42.445717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.445698 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:42.447549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.447525 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-xks-ca" pod "252fabe8-441b-42b5-ad3d-0ad9c29815e6" (UID: "252fabe8-441b-42b5-ad3d-0ad9c29815e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:42.447925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.447595 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252fabe8-441b-42b5-ad3d-0ad9c29815e6-kube-api-access-szlrn" pod "252fabe8-441b-42b5-ad3d-0ad9c29815e6" (UID: "252fabe8-441b-42b5-ad3d-0ad9c29815e6"). InnerVolumeSpecName "kube-api-access-szlrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:42.475928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.475902 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:42.476316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.476297 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:42.477323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.477299 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:35:42.547414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.547381 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szlrn\" (UniqueName: \"kubernetes.io/projected/252fabe8-441b-42b5-ad3d-0ad9c29815e6-kube-api-access-szlrn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:42.547414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:42.547412 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252fabe8-441b-42b5-ad3d-0ad9c29815e6-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:43.169303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:43.169266 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" Nov 02 21:35:43.169477 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:43.169295 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8" event={"ID":"252fabe8-441b-42b5-ad3d-0ad9c29815e6","Type":"ContainerDied","Data":"5af02467b27242fd960338426bc2fbd17c2c07ef6199e86f26883d519d79e25f"} Nov 02 21:35:43.169477 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:43.169347 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af02467b27242fd960338426bc2fbd17c2c07ef6199e86f26883d519d79e25f" Nov 02 21:35:44.868840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:44.868811 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:35:44.869279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:44.868887 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:35:46.084076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:46.084040 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:46.378740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:46.378648 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:46.604661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:46.604608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:47.024471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.024423 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr"] Nov 02 21:35:47.025880 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.025851 4441 cpu_manager.go:414] "containerMap: removing container" podUID="252fabe8-441b-42b5-ad3d-0ad9c29815e6" containerName="revoke-keys" Nov 02 21:35:47.026049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.026034 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252fabe8-441b-42b5-ad3d-0ad9c29815e6" containerName="revoke-keys" Nov 02 21:35:47.026364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.026350 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="252fabe8-441b-42b5-ad3d-0ad9c29815e6" containerName="revoke-keys" Nov 02 21:35:47.084267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.084238 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.181502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.181469 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr"] Nov 02 21:35:47.183733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.183701 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsnt\" (UniqueName: \"kubernetes.io/projected/c5519e76-3d13-4b6e-b751-2d0f09c2244a-kube-api-access-4vsnt\") pod \"payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.184017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.183994 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5519e76-3d13-4b6e-b751-2d0f09c2244a-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.223570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.222550 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c7cd2d37\"" Nov 02 21:35:47.223570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.222618 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c7cd2d37-dockercfg-62gwz\"" Nov 02 21:35:47.285959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.285865 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsnt\" (UniqueName: \"kubernetes.io/projected/c5519e76-3d13-4b6e-b751-2d0f09c2244a-kube-api-access-4vsnt\") pod \"payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.286161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.286004 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5519e76-3d13-4b6e-b751-2d0f09c2244a-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.287081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.287054 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5519e76-3d13-4b6e-b751-2d0f09c2244a-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.339715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.339684 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsnt\" (UniqueName: \"kubernetes.io/projected/c5519e76-3d13-4b6e-b751-2d0f09c2244a-kube-api-access-4vsnt\") pod \"payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.501466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.501431 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:47.892237 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:47.892155 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr"] Nov 02 21:35:47.895802 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:47.895762 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5519e76_3d13_4b6e_b751_2d0f09c2244a.slice/crio-2eca53f29c3ac9da4bcd59950c14655b3b2497e7ac7adb35bff43caef5585dbe WatchSource:0}: Error finding container 2eca53f29c3ac9da4bcd59950c14655b3b2497e7ac7adb35bff43caef5585dbe: Status 404 returned error can't find the container with id 2eca53f29c3ac9da4bcd59950c14655b3b2497e7ac7adb35bff43caef5585dbe Nov 02 21:35:48.130265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:48.130232 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:48.249630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:48.249529 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" event={"ID":"c5519e76-3d13-4b6e-b751-2d0f09c2244a","Type":"ContainerStarted","Data":"9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5"} Nov 02 21:35:48.249630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:48.249594 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" event={"ID":"c5519e76-3d13-4b6e-b751-2d0f09c2244a","Type":"ContainerStarted","Data":"2eca53f29c3ac9da4bcd59950c14655b3b2497e7ac7adb35bff43caef5585dbe"} Nov 02 21:35:48.249630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:48.249627 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:48.360742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:48.360694 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" podStartSLOduration=2.360677548 podStartE2EDuration="2.360677548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:48.356725235 +0000 UTC m=+1377.912353609" watchObservedRunningTime="2026-11-02 21:35:48.360677548 +0000 UTC m=+1377.916305886" Nov 02 21:35:48.479562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:48.479527 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:51.043560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.043524 4441 scope.go:122] "RemoveContainer" containerID="78e49ffba4f2a6f58df156bdfeeb6d0c6df3c21dc11bb4af33fb9bf0b56bed89" Nov 02 21:35:51.056046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.056026 4441 scope.go:122] "RemoveContainer" containerID="236a3ec0992281c0f818150aaff59f92269874c30d73f0e061c53274d55bd6dc" Nov 02 21:35:51.069572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.069552 4441 scope.go:122] "RemoveContainer" containerID="ad93625c60fef6a87dcd783546c6f1583b6f9d2da3761bbcb78ea3c8da63c65c" Nov 02 21:35:51.082431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.082406 4441 scope.go:122] "RemoveContainer" containerID="0ae11528cf995d7c87430540434b29e10d7c050458cd829632842fe412451546" Nov 02 21:35:51.092827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.092804 4441 scope.go:122] "RemoveContainer" containerID="600ecfa5a9ed5b2860ff7ea624baaa35039ebbb5de317eb7085d34914d3ed2a8" Nov 02 21:35:51.103330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.103306 4441 scope.go:122] "RemoveContainer" containerID="919400fbf8aa9e4f9f5c68bbfdfbdfb231ff622a3d32a0395771920bc72953df" Nov 02 21:35:51.119035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.119011 4441 scope.go:122] "RemoveContainer" containerID="a033b930fb4080abaf1255b3e0f4ac9a74fbf8e4f39ea8789b4e6f6bc127ad4e" Nov 02 21:35:51.129510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.129485 4441 scope.go:122] "RemoveContainer" containerID="ba520c7609ce434daad497d0a77af5b01a641042120fc44a444a0520400ef15f" Nov 02 21:35:51.139463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.139443 4441 scope.go:122] "RemoveContainer" containerID="fc3bb737321bf44e7c7d596b041008d26de340df307130e53feac836474cd30d" Nov 02 21:35:51.148918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.148896 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:35:51.150940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.150923 4441 scope.go:122] "RemoveContainer" containerID="6b15fa02a12f956ced4f79bffff513600cca5beb517c0338f667f987c3658edc" Nov 02 21:35:51.168332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.168285 4441 scope.go:122] "RemoveContainer" containerID="31ccb6d13c1119a96a119e3fa8eec882fff128cd1921e689818c274793a9dce5" Nov 02 21:35:51.186494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.186471 4441 scope.go:122] "RemoveContainer" containerID="dadcda358b286a08b8cd5f5c6142bb6a1a8f55cef50b0e6411824fa534b03620" Nov 02 21:35:51.201831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.201809 4441 scope.go:122] "RemoveContainer" containerID="ac9f03f03552ee560b9089d45760a3258badb1d81c36416d56738e82177f0521" Nov 02 21:35:51.312018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.300311 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:51.322168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.321484 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7"] Nov 02 21:35:51.331043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.330962 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.384778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.384747 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 21:35:51.384947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.384904 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 21:35:51.385051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.384950 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 21:35:51.385616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.385590 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nwm6f\"" Nov 02 21:35:51.388007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.385833 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7"] Nov 02 21:35:51.432682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.432608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:51.491986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.490870 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plch2\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-kube-api-access-plch2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.491986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.490958 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.491986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.491008 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.491986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.491035 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.491986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.491184 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.491986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.491269 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9949816-0828-42b1-831a-27f618456cf1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.491986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.491357 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.593452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.593396 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.593653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.593491 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9949816-0828-42b1-831a-27f618456cf1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.593653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.593574 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.593792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.593749 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plch2\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-kube-api-access-plch2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.594169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.594142 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.594495 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.594471 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.594579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.594548 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.619208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.610648 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.641018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.640654 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.642337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.642283 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9949816-0828-42b1-831a-27f618456cf1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.642474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.642358 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.642781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.642759 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.647568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.647486 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plch2\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-kube-api-access-plch2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.659080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.658991 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m56v7\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:51.955037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:51.952260 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:35:53.108522 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:35:53.108422 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9949816_0828_42b1_831a_27f618456cf1.slice/crio-4c17678124d764fd94e67cbe386ae4bb2d3f61d24599a58acbc1ad02ec1e452c WatchSource:0}: Error finding container 4c17678124d764fd94e67cbe386ae4bb2d3f61d24599a58acbc1ad02ec1e452c: Status 404 returned error can't find the container with id 4c17678124d764fd94e67cbe386ae4bb2d3f61d24599a58acbc1ad02ec1e452c Nov 02 21:35:53.198389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:53.194281 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:35:53.271991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:53.268615 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7"] Nov 02 21:35:54.047927 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.047394 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.179010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.173932 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:35:54.179010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.174326 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.281992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.278451 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" event={"ID":"f9949816-0828-42b1-831a-27f618456cf1","Type":"ContainerStarted","Data":"4c17678124d764fd94e67cbe386ae4bb2d3f61d24599a58acbc1ad02ec1e452c"} Nov 02 21:35:54.295003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.288248 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.369603 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.369556 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.370093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.370048 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.575133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.574049 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.575133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.574803 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:35:54.610041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.575417 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.646394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.610697 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.684993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.647443 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:54.684993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.677054 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.684993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.677500 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 02 21:35:54.742460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:54.742408 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.051121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.051007 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.074185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.074108 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.313950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.308245 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 02 21:35:55.313950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.308410 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:35:55.314528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.314102 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.338011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.337029 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.404205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.404163 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.468995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.416036 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.579122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.575404 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.583154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.583114 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:55.627023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.615773 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.628305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.628125 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658464 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658610 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658684 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658743 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658800 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658854 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658921 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.658996 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:35:55.664017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.659047 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:35:55.700014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.690999 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2k8gl" podUID="ace09442-f99c-432d-8f6a-995c68786fb0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.34:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.700628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.700572 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:35:55.702332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.700881 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.704090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.704071 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:35:55.705507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.705488 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:35:55.808866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.756477 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.823990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.809130 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:35:55.937140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.912116 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:35:55.937140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.912255 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.975022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:55.974984 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tng69" podUID="30f6edef-f667-44fa-8cea-f87ba914b7b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.108391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.102413 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.111185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.111146 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 02 21:35:56.113577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.113556 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:35:56.255112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.255026 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:35:56.275380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.265572 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:35:56.337205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.333323 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.351361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.351311 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.355438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.355398 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:35:56.367818 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.367789 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:35:56.410021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.409916 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.417040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.413029 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:35:56.417040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.416299 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:35:56.463039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.458603 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:35:56.465737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.465716 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:35:56.475448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.475428 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:35:56.483304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.483056 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:35:56.493487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.493468 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:35:56.493634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:56.493623 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:35:58.004023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:58.003956 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:59.262598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:59.262563 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:35:59.402664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:59.402628 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:59.402817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:59.402695 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:35:59.880902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:59.880863 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" event={"ID":"f9949816-0828-42b1-831a-27f618456cf1","Type":"ContainerStarted","Data":"5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432"} Nov 02 21:35:59.881112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:35:59.880924 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:36:00.013937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:00.013841 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" podStartSLOduration=2.807426359 podStartE2EDuration="9.013822356s" totalImagesPullingTime="6.206395997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:00.013051996 +0000 UTC m=+1389.568680345" watchObservedRunningTime="2026-11-02 21:36:00.013822356 +0000 UTC m=+1389.569450695" Nov 02 21:36:00.943414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:00.943070 4441 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-m56v7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:36:00.943414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:00.943132 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" podUID="f9949816-0828-42b1-831a-27f618456cf1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:36:02.789923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:02.789842 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:03.942720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:03.942687 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:36:04.638206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:04.638173 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:07.244745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:07.244696 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:07.394824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:07.394776 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:07.522486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:07.522371 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:10.622200 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:36:10.622164 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:36:11.940464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:11.940418 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:12.091565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:12.091532 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:12.225521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:12.225317 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:16.508026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:16.507995 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:21.082888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:21.082855 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:21.229651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:21.229621 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:21.356429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:21.356325 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:25.725827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:25.725790 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:30.423204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:30.423167 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:35.063838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:35.063804 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:35.213065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:35.213034 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:35.329524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:35.329442 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:40.414013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:40.413944 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:40.746026 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:36:40.745889 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:36:42.138563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.138532 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql"] Nov 02 21:36:42.143318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.143298 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195118 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195161 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195309 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqpk\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-kube-api-access-qzqpk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195361 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80264ee9-9160-4b14-a5f8-c4938714fac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195501 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195594 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195615 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195631 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.195724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.195658 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.250872 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.250835 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql"] Nov 02 21:36:42.310382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.310341 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqpk\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-kube-api-access-qzqpk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312111 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80264ee9-9160-4b14-a5f8-c4938714fac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312224 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312294 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312323 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312348 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312384 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312503 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.312601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.312544 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.313124 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.313091 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.315020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.314536 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.315020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.314575 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.315020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.314957 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80264ee9-9160-4b14-a5f8-c4938714fac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.315336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.315316 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.319357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.317650 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.321301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.320606 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.365898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.365869 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqpk\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-kube-api-access-qzqpk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.366773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.366756 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7g2ql\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:42.458863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:42.458789 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:43.016592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.016328 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql"] Nov 02 21:36:43.023466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.023422 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:36:43.023713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.023689 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:36:43.023889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.023871 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:36:43.433205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.433166 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" event={"ID":"80264ee9-9160-4b14-a5f8-c4938714fac4","Type":"ContainerStarted","Data":"3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377"} Nov 02 21:36:43.433707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.433227 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" event={"ID":"80264ee9-9160-4b14-a5f8-c4938714fac4","Type":"ContainerStarted","Data":"851a5bbd5651d032447086727bb2e26d26ac4a49edac98a911ceef812fb82e38"} Nov 02 21:36:43.459001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.458952 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:43.545698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:43.545641 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podStartSLOduration=1.545623985 podStartE2EDuration="1.545623985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:43.542942115 +0000 UTC m=+1433.098570452" watchObservedRunningTime="2026-11-02 21:36:43.545623985 +0000 UTC m=+1433.101252323" Nov 02 21:36:44.461178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:44.461133 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:44.461780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:44.461749 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:45.303666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:45.303624 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:45.480052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:45.476108 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:45.480052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:45.476165 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:46.459151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:46.459117 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:46.459322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:46.459172 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:47.459625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:47.459586 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:47.460082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:47.459650 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:48.458524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:48.458490 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:48.459600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:48.459571 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:48.459730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:48.459620 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:48.910153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:48.909320 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:49.459196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:49.459153 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:49.459365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:49.459231 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:49.794730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:49.794645 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:50.459320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.459232 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:50.459320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.459280 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:50.564461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.564419 4441 generic.go:356] "Generic (PLEG): container finished" podID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerID="99acbefa13ac9e1aa413d2dfa618f790376b09692a9dd4d806be42af73850761" exitCode=0 Nov 02 21:36:50.564624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.564489 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" event={"ID":"dc184427-21af-49cc-b2fd-07f5426aedaf","Type":"ContainerDied","Data":"99acbefa13ac9e1aa413d2dfa618f790376b09692a9dd4d806be42af73850761"} Nov 02 21:36:50.564873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.564858 4441 scope.go:122] "RemoveContainer" containerID="99acbefa13ac9e1aa413d2dfa618f790376b09692a9dd4d806be42af73850761" Nov 02 21:36:50.566301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.566272 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:36:50.566408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.566354 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:36:50.566408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:50.566401 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:36:51.358858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.358831 4441 scope.go:122] "RemoveContainer" containerID="fecdcac34ac6ba2f9887a0b1fa4ff214848b2efe93a77c8e14a27f5c26b641cb" Nov 02 21:36:51.409888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.409855 4441 scope.go:122] "RemoveContainer" containerID="5f2916631b8ff18f7154d081634004fff5bfe53361403cd1f0727f4cc99cd08a" Nov 02 21:36:51.437447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.437363 4441 scope.go:122] "RemoveContainer" containerID="f1caa4aa32d021a121d016c8b0180ba1d19eebc1083c3e7a72d92bef7794253f" Nov 02 21:36:51.459188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.459168 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 21:36:51.459283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.459212 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 21:36:51.605946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.605909 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" event={"ID":"dc184427-21af-49cc-b2fd-07f5426aedaf","Type":"ContainerStarted","Data":"086035d08106eb970e559613b826083c7018f892c51ea5316d095ddf862ba1a6"} Nov 02 21:36:51.764851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.764763 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:36:51.765040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.765019 4441 patch_prober.go:29] interesting pod/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:36:51.765117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:51.765079 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 21:36:52.764864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:52.764831 4441 patch_prober.go:29] interesting pod/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:36:52.765352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:52.764893 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 21:36:52.861619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:52.861584 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:52.862545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:52.862511 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:52.863321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:52.863290 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:36:53.221176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.221128 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k"] Nov 02 21:36:53.228378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.228352 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.268306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.268278 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k"] Nov 02 21:36:53.304004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.301434 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:36:53.304416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.304392 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:36:53.312193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.312167 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:36:53.312320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.312239 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:36:53.349752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.349721 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lcfg\" (UniqueName: \"kubernetes.io/projected/a2a82f95-7b03-415c-81ee-11a3500fb4f5-kube-api-access-9lcfg\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.349913 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.349786 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.349913 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.349896 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.451714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.451681 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lcfg\" (UniqueName: \"kubernetes.io/projected/a2a82f95-7b03-415c-81ee-11a3500fb4f5-kube-api-access-9lcfg\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.451916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.451761 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.451916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.451844 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.452591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.452572 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.454181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.454163 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.505301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.505226 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lcfg\" (UniqueName: \"kubernetes.io/projected/a2a82f95-7b03-415c-81ee-11a3500fb4f5-kube-api-access-9lcfg\") pod \"maas-api-revoke-sub-bf03a9d9-nlm5k\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.544096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.544068 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:53.764553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.764470 4441 patch_prober.go:29] interesting pod/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:36:53.764553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:53.764534 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 21:36:54.004088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.004052 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k"] Nov 02 21:36:54.005593 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:36:54.005564 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a82f95_7b03_415c_81ee_11a3500fb4f5.slice/crio-732ae2681334db96e24f48bd173c8a7549213f1418d8b24ddd4696a8f39c9588 WatchSource:0}: Error finding container 732ae2681334db96e24f48bd173c8a7549213f1418d8b24ddd4696a8f39c9588: Status 404 returned error can't find the container with id 732ae2681334db96e24f48bd173c8a7549213f1418d8b24ddd4696a8f39c9588 Nov 02 21:36:54.409244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.409212 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:54.530507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.530476 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:54.657621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.657536 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" event={"ID":"a2a82f95-7b03-415c-81ee-11a3500fb4f5","Type":"ContainerStarted","Data":"cf0e0a6fc2e1b8ddb4999e722e9f311ffc9eb84e7a361ccbc4a8beed27516a1c"} Nov 02 21:36:54.657621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.657593 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" event={"ID":"a2a82f95-7b03-415c-81ee-11a3500fb4f5","Type":"ContainerStarted","Data":"732ae2681334db96e24f48bd173c8a7549213f1418d8b24ddd4696a8f39c9588"} Nov 02 21:36:54.764820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.764786 4441 patch_prober.go:29] interesting pod/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:36:54.764993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.764839 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 21:36:54.805336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:54.799073 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" podStartSLOduration=1.799049996 podStartE2EDuration="1.799049996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:54.767214344 +0000 UTC m=+1444.322842681" watchObservedRunningTime="2026-11-02 21:36:54.799049996 +0000 UTC m=+1444.354678334" Nov 02 21:36:55.699468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:55.699433 4441 generic.go:356] "Generic (PLEG): container finished" podID="a2a82f95-7b03-415c-81ee-11a3500fb4f5" containerID="cf0e0a6fc2e1b8ddb4999e722e9f311ffc9eb84e7a361ccbc4a8beed27516a1c" exitCode=0 Nov 02 21:36:55.699886 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:55.699471 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" event={"ID":"a2a82f95-7b03-415c-81ee-11a3500fb4f5","Type":"ContainerDied","Data":"cf0e0a6fc2e1b8ddb4999e722e9f311ffc9eb84e7a361ccbc4a8beed27516a1c"} Nov 02 21:36:55.765279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:55.765244 4441 patch_prober.go:29] interesting pod/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:36:55.765432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:55.765303 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 21:36:56.765397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.765357 4441 patch_prober.go:29] interesting pod/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:36:56.765758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.765426 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 21:36:56.892424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.892399 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:56.928235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.928201 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a82f95-7b03-415c-81ee-11a3500fb4f5-kube-api-access-9lcfg\" (UniqueName: \"kubernetes.io/projected/a2a82f95-7b03-415c-81ee-11a3500fb4f5-kube-api-access-9lcfg\") pod \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " Nov 02 21:36:56.928380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.928314 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-xks-ca\") pod \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " Nov 02 21:36:56.928380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.928346 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-openshift-ca\") pod \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\" (UID: \"a2a82f95-7b03-415c-81ee-11a3500fb4f5\") " Nov 02 21:36:56.928758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.928729 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-openshift-ca" pod "a2a82f95-7b03-415c-81ee-11a3500fb4f5" (UID: "a2a82f95-7b03-415c-81ee-11a3500fb4f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:56.929139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.929118 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:56.930317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.930288 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-xks-ca" pod "a2a82f95-7b03-415c-81ee-11a3500fb4f5" (UID: "a2a82f95-7b03-415c-81ee-11a3500fb4f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:56.930408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:56.930389 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a82f95-7b03-415c-81ee-11a3500fb4f5-kube-api-access-9lcfg" pod "a2a82f95-7b03-415c-81ee-11a3500fb4f5" (UID: "a2a82f95-7b03-415c-81ee-11a3500fb4f5"). InnerVolumeSpecName "kube-api-access-9lcfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:57.031043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.030945 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a82f95-7b03-415c-81ee-11a3500fb4f5-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:57.031043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.030990 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lcfg\" (UniqueName: \"kubernetes.io/projected/a2a82f95-7b03-415c-81ee-11a3500fb4f5-kube-api-access-9lcfg\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:57.635589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.635557 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:36:57.722426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.722389 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" event={"ID":"a2a82f95-7b03-415c-81ee-11a3500fb4f5","Type":"ContainerDied","Data":"732ae2681334db96e24f48bd173c8a7549213f1418d8b24ddd4696a8f39c9588"} Nov 02 21:36:57.722592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.722469 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732ae2681334db96e24f48bd173c8a7549213f1418d8b24ddd4696a8f39c9588" Nov 02 21:36:57.722592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.722398 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k" Nov 02 21:36:57.769106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.769081 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:36:57.770346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:57.770321 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:36:58.925808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:36:58.925772 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:03.419801 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:03.419768 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:03.908078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:03.908043 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:08.177751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:08.177719 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:10.004502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.004461 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:10.203558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.203523 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7"] Nov 02 21:37:10.204658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.204637 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a2a82f95-7b03-415c-81ee-11a3500fb4f5" containerName="revoke-keys" Nov 02 21:37:10.204716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.204661 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a82f95-7b03-415c-81ee-11a3500fb4f5" containerName="revoke-keys" Nov 02 21:37:10.204863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.204851 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a82f95-7b03-415c-81ee-11a3500fb4f5" containerName="revoke-keys" Nov 02 21:37:10.209513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.209489 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.243351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.243327 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjg2r\" (UniqueName: \"kubernetes.io/projected/4393120d-4ac6-4690-b325-a235aaa40df3-kube-api-access-xjg2r\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.243473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.243406 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.243562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.243548 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.256117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.256061 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7"] Nov 02 21:37:10.259197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.259173 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:37:10.296218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.296190 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:37:10.296435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.296298 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:37:10.306560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.306535 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:37:10.344522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.344492 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjg2r\" (UniqueName: \"kubernetes.io/projected/4393120d-4ac6-4690-b325-a235aaa40df3-kube-api-access-xjg2r\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.344651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.344602 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.344831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.344800 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.345431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.345409 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.347456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.347429 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.398511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.398480 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjg2r\" (UniqueName: \"kubernetes.io/projected/4393120d-4ac6-4690-b325-a235aaa40df3-kube-api-access-xjg2r\") pod \"maas-api-revoke-keys-e2e-ait-c39c4123-547v7\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.525123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.525048 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:10.617037 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:37:10.616954 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:37:10.946426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:10.946399 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7"] Nov 02 21:37:10.949113 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:37:10.949086 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4393120d_4ac6_4690_b325_a235aaa40df3.slice/crio-700ca289c9dc1179c49d5c2a146db4d896c59868bf9fbf2569e7985e0b4e17fb WatchSource:0}: Error finding container 700ca289c9dc1179c49d5c2a146db4d896c59868bf9fbf2569e7985e0b4e17fb: Status 404 returned error can't find the container with id 700ca289c9dc1179c49d5c2a146db4d896c59868bf9fbf2569e7985e0b4e17fb Nov 02 21:37:11.864901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:11.864868 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" event={"ID":"4393120d-4ac6-4690-b325-a235aaa40df3","Type":"ContainerStarted","Data":"8ad5aaa82c398e5631f8a2265d9f0473c209aeaf534321c1681c667920ebfdce"} Nov 02 21:37:11.865276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:11.864930 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" event={"ID":"4393120d-4ac6-4690-b325-a235aaa40df3","Type":"ContainerStarted","Data":"700ca289c9dc1179c49d5c2a146db4d896c59868bf9fbf2569e7985e0b4e17fb"} Nov 02 21:37:11.975508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:11.975464 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" podStartSLOduration=1.975449828 podStartE2EDuration="1.975449828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:11.971875617 +0000 UTC m=+1461.527503968" watchObservedRunningTime="2026-11-02 21:37:11.975449828 +0000 UTC m=+1461.531078180" Nov 02 21:37:12.382226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.382191 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4"] Nov 02 21:37:12.393550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.393516 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.475703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.475662 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.475922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.475732 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.475922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.475855 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmqx\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-kube-api-access-hzmqx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.475922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.475882 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.476176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.475986 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.476176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.476078 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.476176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.476101 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.476402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.476225 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.476402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.476250 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.517173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.517127 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4"] Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578411 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578455 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578554 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578579 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578634 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578678 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578789 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmqx\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-kube-api-access-hzmqx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578820 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.578906 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.579884 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.580672 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.580909 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.581583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.581516 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.582680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.582658 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.582859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.582840 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.585088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.585066 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.592050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.592026 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:12.632446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.632382 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.634781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.634760 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmqx\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-kube-api-access-hzmqx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kgm4\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.716358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.716319 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:12.883581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.882752 4441 generic.go:356] "Generic (PLEG): container finished" podID="4393120d-4ac6-4690-b325-a235aaa40df3" containerID="8ad5aaa82c398e5631f8a2265d9f0473c209aeaf534321c1681c667920ebfdce" exitCode=0 Nov 02 21:37:12.883581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:12.883021 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" event={"ID":"4393120d-4ac6-4690-b325-a235aaa40df3","Type":"ContainerDied","Data":"8ad5aaa82c398e5631f8a2265d9f0473c209aeaf534321c1681c667920ebfdce"} Nov 02 21:37:13.177958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:13.177933 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4"] Nov 02 21:37:13.180411 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:37:13.180379 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a378a3_f8dc_48e5_baf8_bb02239ec7ba.slice/crio-5102aad8f4ce1c36cbd4266ffac1c11084a6bd7429ae89b00bbaee033ea74c49 WatchSource:0}: Error finding container 5102aad8f4ce1c36cbd4266ffac1c11084a6bd7429ae89b00bbaee033ea74c49: Status 404 returned error can't find the container with id 5102aad8f4ce1c36cbd4266ffac1c11084a6bd7429ae89b00bbaee033ea74c49 Nov 02 21:37:13.183402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:13.183367 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:13.183516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:13.183453 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:13.183516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:13.183503 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:13.897589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:13.897555 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" event={"ID":"60a378a3-f8dc-48e5-baf8-bb02239ec7ba","Type":"ContainerStarted","Data":"061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21"} Nov 02 21:37:13.899225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:13.898592 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" event={"ID":"60a378a3-f8dc-48e5-baf8-bb02239ec7ba","Type":"ContainerStarted","Data":"5102aad8f4ce1c36cbd4266ffac1c11084a6bd7429ae89b00bbaee033ea74c49"} Nov 02 21:37:14.065138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.065069 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podStartSLOduration=2.065048413 podStartE2EDuration="2.065048413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:14.011411599 +0000 UTC m=+1463.567039936" watchObservedRunningTime="2026-11-02 21:37:14.065048413 +0000 UTC m=+1463.620676751" Nov 02 21:37:14.114768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.114738 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:14.206555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.206454 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-xks-ca\") pod \"4393120d-4ac6-4690-b325-a235aaa40df3\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " Nov 02 21:37:14.206758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.206549 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4393120d-4ac6-4690-b325-a235aaa40df3-kube-api-access-xjg2r\" (UniqueName: \"kubernetes.io/projected/4393120d-4ac6-4690-b325-a235aaa40df3-kube-api-access-xjg2r\") pod \"4393120d-4ac6-4690-b325-a235aaa40df3\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " Nov 02 21:37:14.206758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.206675 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-openshift-ca\") pod \"4393120d-4ac6-4690-b325-a235aaa40df3\" (UID: \"4393120d-4ac6-4690-b325-a235aaa40df3\") " Nov 02 21:37:14.207109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.207086 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-openshift-ca" pod "4393120d-4ac6-4690-b325-a235aaa40df3" (UID: "4393120d-4ac6-4690-b325-a235aaa40df3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:14.208896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.208873 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:14.208896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.208885 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-xks-ca" pod "4393120d-4ac6-4690-b325-a235aaa40df3" (UID: "4393120d-4ac6-4690-b325-a235aaa40df3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:14.209308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.209277 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4393120d-4ac6-4690-b325-a235aaa40df3-kube-api-access-xjg2r" pod "4393120d-4ac6-4690-b325-a235aaa40df3" (UID: "4393120d-4ac6-4690-b325-a235aaa40df3"). InnerVolumeSpecName "kube-api-access-xjg2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:14.310650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.310593 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4393120d-4ac6-4690-b325-a235aaa40df3-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:14.310862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.310650 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjg2r\" (UniqueName: \"kubernetes.io/projected/4393120d-4ac6-4690-b325-a235aaa40df3-kube-api-access-xjg2r\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:14.717599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.717558 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:14.718944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.718916 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:14.719080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.718979 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:14.914943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.914182 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" Nov 02 21:37:14.914943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.914207 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7" event={"ID":"4393120d-4ac6-4690-b325-a235aaa40df3","Type":"ContainerDied","Data":"700ca289c9dc1179c49d5c2a146db4d896c59868bf9fbf2569e7985e0b4e17fb"} Nov 02 21:37:14.914943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:14.914258 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="700ca289c9dc1179c49d5c2a146db4d896c59868bf9fbf2569e7985e0b4e17fb" Nov 02 21:37:15.416518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.416471 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx"] Nov 02 21:37:15.417282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.417261 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4393120d-4ac6-4690-b325-a235aaa40df3" containerName="revoke-keys" Nov 02 21:37:15.417282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.417281 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4393120d-4ac6-4690-b325-a235aaa40df3" containerName="revoke-keys" Nov 02 21:37:15.417517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.417502 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4393120d-4ac6-4690-b325-a235aaa40df3" containerName="revoke-keys" Nov 02 21:37:15.421742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.421719 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.537066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.536921 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:37:15.537066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.537058 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:37:15.537342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.537323 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:37:15.546832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.537916 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:37:15.546832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.539109 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx"] Nov 02 21:37:15.639072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.639040 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7nw\" (UniqueName: \"kubernetes.io/projected/9bd19779-7474-42c0-a057-5a1d7e6d2746-kube-api-access-7h7nw\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.639315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.639294 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.639354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.639331 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.717636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.717555 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:15.717636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.717603 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:15.740553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.740519 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.740691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.740558 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.740771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.740685 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7nw\" (UniqueName: \"kubernetes.io/projected/9bd19779-7474-42c0-a057-5a1d7e6d2746-kube-api-access-7h7nw\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.741310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.741281 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.742854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.742832 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:15.803585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:15.803538 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7nw\" (UniqueName: \"kubernetes.io/projected/9bd19779-7474-42c0-a057-5a1d7e6d2746-kube-api-access-7h7nw\") pod \"maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:16.041892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:16.041801 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:16.465056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:16.465025 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx"] Nov 02 21:37:16.481236 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:37:16.469003 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd19779_7474_42c0_a057_5a1d7e6d2746.slice/crio-4ae6059ec5b34ffd330ed2c79dfadb189c7c76540b0125f028ad78a5fcf40b9d WatchSource:0}: Error finding container 4ae6059ec5b34ffd330ed2c79dfadb189c7c76540b0125f028ad78a5fcf40b9d: Status 404 returned error can't find the container with id 4ae6059ec5b34ffd330ed2c79dfadb189c7c76540b0125f028ad78a5fcf40b9d Nov 02 21:37:16.717563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:16.717500 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:16.717563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:16.717553 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:16.935768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:16.935734 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" event={"ID":"9bd19779-7474-42c0-a057-5a1d7e6d2746","Type":"ContainerStarted","Data":"df8938677ad54d2af71dd63bc37d552f12685fd84eeb6d5085e4fb06009df330"} Nov 02 21:37:16.935919 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:16.935794 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" event={"ID":"9bd19779-7474-42c0-a057-5a1d7e6d2746","Type":"ContainerStarted","Data":"4ae6059ec5b34ffd330ed2c79dfadb189c7c76540b0125f028ad78a5fcf40b9d"} Nov 02 21:37:17.052123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:17.052025 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" podStartSLOduration=2.052010869 podStartE2EDuration="2.052010869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:17.046750867 +0000 UTC m=+1466.602379214" watchObservedRunningTime="2026-11-02 21:37:17.052010869 +0000 UTC m=+1466.607639206" Nov 02 21:37:17.707183 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:37:17.707144 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd19779_7474_42c0_a057_5a1d7e6d2746.slice/crio-conmon-df8938677ad54d2af71dd63bc37d552f12685fd84eeb6d5085e4fb06009df330.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:37:17.717839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:17.717799 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:17.718011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:17.717860 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:17.730139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:17.730105 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:17.949715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:17.949624 4441 generic.go:356] "Generic (PLEG): container finished" podID="9bd19779-7474-42c0-a057-5a1d7e6d2746" containerID="df8938677ad54d2af71dd63bc37d552f12685fd84eeb6d5085e4fb06009df330" exitCode=0 Nov 02 21:37:17.949875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:17.949721 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" event={"ID":"9bd19779-7474-42c0-a057-5a1d7e6d2746","Type":"ContainerDied","Data":"df8938677ad54d2af71dd63bc37d552f12685fd84eeb6d5085e4fb06009df330"} Nov 02 21:37:18.717531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:18.717486 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:18.718062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:18.717540 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:19.213797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.213772 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:19.322197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.322160 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-xks-ca\") pod \"9bd19779-7474-42c0-a057-5a1d7e6d2746\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " Nov 02 21:37:19.322465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.322447 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-openshift-ca\") pod \"9bd19779-7474-42c0-a057-5a1d7e6d2746\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " Nov 02 21:37:19.322574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.322559 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd19779-7474-42c0-a057-5a1d7e6d2746-kube-api-access-7h7nw\" (UniqueName: \"kubernetes.io/projected/9bd19779-7474-42c0-a057-5a1d7e6d2746-kube-api-access-7h7nw\") pod \"9bd19779-7474-42c0-a057-5a1d7e6d2746\" (UID: \"9bd19779-7474-42c0-a057-5a1d7e6d2746\") " Nov 02 21:37:19.323037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.322882 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-openshift-ca" pod "9bd19779-7474-42c0-a057-5a1d7e6d2746" (UID: "9bd19779-7474-42c0-a057-5a1d7e6d2746"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:19.323369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.323342 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:19.326098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.326073 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd19779-7474-42c0-a057-5a1d7e6d2746-kube-api-access-7h7nw" pod "9bd19779-7474-42c0-a057-5a1d7e6d2746" (UID: "9bd19779-7474-42c0-a057-5a1d7e6d2746"). InnerVolumeSpecName "kube-api-access-7h7nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:19.326589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.326566 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-xks-ca" pod "9bd19779-7474-42c0-a057-5a1d7e6d2746" (UID: "9bd19779-7474-42c0-a057-5a1d7e6d2746"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:19.409990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.409789 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x"] Nov 02 21:37:19.410720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.410697 4441 cpu_manager.go:414] "containerMap: removing container" podUID="9bd19779-7474-42c0-a057-5a1d7e6d2746" containerName="revoke-keys" Nov 02 21:37:19.410720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.410718 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd19779-7474-42c0-a057-5a1d7e6d2746" containerName="revoke-keys" Nov 02 21:37:19.410889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.410873 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd19779-7474-42c0-a057-5a1d7e6d2746" containerName="revoke-keys" Nov 02 21:37:19.427985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.417528 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.432915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.432885 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7nw\" (UniqueName: \"kubernetes.io/projected/9bd19779-7474-42c0-a057-5a1d7e6d2746-kube-api-access-7h7nw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:19.434455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.434428 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd19779-7474-42c0-a057-5a1d7e6d2746-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:19.442627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.442602 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x"] Nov 02 21:37:19.536091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.536051 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjh5d\" (UniqueName: \"kubernetes.io/projected/f710b209-cf07-4058-9034-229244973651-kube-api-access-sjh5d\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.536340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.536313 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f710b209-cf07-4058-9034-229244973651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.536509 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.536474 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f710b209-cf07-4058-9034-229244973651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.638406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.638317 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjh5d\" (UniqueName: \"kubernetes.io/projected/f710b209-cf07-4058-9034-229244973651-kube-api-access-sjh5d\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.638605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.638582 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f710b209-cf07-4058-9034-229244973651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.638722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.638705 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f710b209-cf07-4058-9034-229244973651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.639342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.639318 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f710b209-cf07-4058-9034-229244973651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.641267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.641240 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f710b209-cf07-4058-9034-229244973651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.692307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.692274 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjh5d\" (UniqueName: \"kubernetes.io/projected/f710b209-cf07-4058-9034-229244973651-kube-api-access-sjh5d\") pod \"maas-api-revoke-sub-360f33f8-h927x\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.717264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.717228 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:19.717396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.717295 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:19.746676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.746647 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:19.970849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.970755 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" event={"ID":"9bd19779-7474-42c0-a057-5a1d7e6d2746","Type":"ContainerDied","Data":"4ae6059ec5b34ffd330ed2c79dfadb189c7c76540b0125f028ad78a5fcf40b9d"} Nov 02 21:37:19.970849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.970786 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx" Nov 02 21:37:19.970849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:19.970806 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae6059ec5b34ffd330ed2c79dfadb189c7c76540b0125f028ad78a5fcf40b9d" Nov 02 21:37:20.170465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:20.170423 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x"] Nov 02 21:37:20.717990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:20.717329 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:20.717990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:20.717394 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:21.001049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:21.000183 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" event={"ID":"f710b209-cf07-4058-9034-229244973651","Type":"ContainerStarted","Data":"c314de03ceda908fa3eaf08bee79b4d2030d9b239ac5bbd5d2925a015b653501"} Nov 02 21:37:21.001049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:21.000255 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" event={"ID":"f710b209-cf07-4058-9034-229244973651","Type":"ContainerStarted","Data":"42b11cb5816d0663e0ac6a22dd6fd110145df11fb255cf4444e4d99cbfbb16e7"} Nov 02 21:37:21.123994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:21.119990 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" podStartSLOduration=2.119955389 podStartE2EDuration="2.119955389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:21.119594152 +0000 UTC m=+1470.675222490" watchObservedRunningTime="2026-11-02 21:37:21.119955389 +0000 UTC m=+1470.675583726" Nov 02 21:37:21.747015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:21.732198 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 21:37:21.747015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:21.732255 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 21:37:21.959304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:21.959202 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55"] Nov 02 21:37:22.040994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:22.019576 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f"] Nov 02 21:37:22.158669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:22.158628 4441 generic.go:356] "Generic (PLEG): container finished" podID="f710b209-cf07-4058-9034-229244973651" containerID="c314de03ceda908fa3eaf08bee79b4d2030d9b239ac5bbd5d2925a015b653501" exitCode=0 Nov 02 21:37:22.159206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:22.159173 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" event={"ID":"f710b209-cf07-4058-9034-229244973651","Type":"ContainerDied","Data":"c314de03ceda908fa3eaf08bee79b4d2030d9b239ac5bbd5d2925a015b653501"} Nov 02 21:37:22.352204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:22.351652 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:22.605048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:22.600537 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:23.818047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:23.783692 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:23.827492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:23.818643 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:37:23.827492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:23.826852 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:24.151573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.151492 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:24.235144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.235092 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f710b209-cf07-4058-9034-229244973651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f710b209-cf07-4058-9034-229244973651-maas-api-openshift-ca\") pod \"f710b209-cf07-4058-9034-229244973651\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " Nov 02 21:37:24.235925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.235902 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f710b209-cf07-4058-9034-229244973651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f710b209-cf07-4058-9034-229244973651-maas-api-xks-ca\") pod \"f710b209-cf07-4058-9034-229244973651\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " Nov 02 21:37:24.236255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.236235 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f710b209-cf07-4058-9034-229244973651-kube-api-access-sjh5d\" (UniqueName: \"kubernetes.io/projected/f710b209-cf07-4058-9034-229244973651-kube-api-access-sjh5d\") pod \"f710b209-cf07-4058-9034-229244973651\" (UID: \"f710b209-cf07-4058-9034-229244973651\") " Nov 02 21:37:24.249011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.235745 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f710b209-cf07-4058-9034-229244973651-maas-api-openshift-ca" pod "f710b209-cf07-4058-9034-229244973651" (UID: "f710b209-cf07-4058-9034-229244973651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:24.249164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.241347 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f710b209-cf07-4058-9034-229244973651-kube-api-access-sjh5d" pod "f710b209-cf07-4058-9034-229244973651" (UID: "f710b209-cf07-4058-9034-229244973651"). InnerVolumeSpecName "kube-api-access-sjh5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:24.261990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.252260 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjh5d\" (UniqueName: \"kubernetes.io/projected/f710b209-cf07-4058-9034-229244973651-kube-api-access-sjh5d\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:24.261990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.252292 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f710b209-cf07-4058-9034-229244973651-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:24.265352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.265321 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f710b209-cf07-4058-9034-229244973651-maas-api-xks-ca" pod "f710b209-cf07-4058-9034-229244973651" (UID: "f710b209-cf07-4058-9034-229244973651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:24.389999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.362265 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f710b209-cf07-4058-9034-229244973651-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:24.567991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.567933 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" event={"ID":"f710b209-cf07-4058-9034-229244973651","Type":"ContainerDied","Data":"42b11cb5816d0663e0ac6a22dd6fd110145df11fb255cf4444e4d99cbfbb16e7"} Nov 02 21:37:24.568280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.568257 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b11cb5816d0663e0ac6a22dd6fd110145df11fb255cf4444e4d99cbfbb16e7" Nov 02 21:37:24.568477 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.568463 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x" Nov 02 21:37:24.827746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.827652 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:37:24.828368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:24.828319 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.119043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.113915 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.214158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.214093 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s"] Nov 02 21:37:25.225995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.215509 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f710b209-cf07-4058-9034-229244973651" containerName="revoke-keys" Nov 02 21:37:25.226454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.226428 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f710b209-cf07-4058-9034-229244973651" containerName="revoke-keys" Nov 02 21:37:25.227079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.227045 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f710b209-cf07-4058-9034-229244973651" containerName="revoke-keys" Nov 02 21:37:25.247319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.247291 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.270142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.270106 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.323842 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5kn\" (UniqueName: \"kubernetes.io/projected/3715402d-8e6d-4348-ae90-e3708cf1960e-kube-api-access-gt5kn\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.323937 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.324108 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.325447 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.352233 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.352347 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.352451 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:37:25.357893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.352508 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:37:25.410048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.409362 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.434991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.427823 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5kn\" (UniqueName: \"kubernetes.io/projected/3715402d-8e6d-4348-ae90-e3708cf1960e-kube-api-access-gt5kn\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.434991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.427907 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.434991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.428058 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.434991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.428345 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.434991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.430632 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.434991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.430049 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.479990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.473045 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:25.506989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.484138 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:37:25.506989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.496476 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s"] Nov 02 21:37:25.534999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.531409 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:25.904514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:25.904455 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5kn\" (UniqueName: \"kubernetes.io/projected/3715402d-8e6d-4348-ae90-e3708cf1960e-kube-api-access-gt5kn\") pod \"maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:26.052765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.052716 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:26.098876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.098841 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:37:26.099215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.099168 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:26.255236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.255129 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:26.258522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.258464 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:26.306000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.275078 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:26.306000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.302997 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:37:26.437447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.433999 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:26.437447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.434366 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:37:26.457199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.448126 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:26.457199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.448495 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:37:26.962273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.946119 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:27.026716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:26.989326 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:37:27.127378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.010332 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" podUID="6a3865cb-8635-4023-bbe9-632c2095371a" containerName="payload-processing" containerID="cri-o://30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81" gracePeriod=25 Nov 02 21:37:27.127378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.043048 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" podUID="820029e7-2538-44c4-bd43-a3ad4bae01b3" containerName="payload-pre-processing" containerID="cri-o://9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a" gracePeriod=25 Nov 02 21:37:27.158600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.158526 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:27.265631 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.260877 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:27.266557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.264578 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:37:27.266778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.266527 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:27.267508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.267490 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:37:27.448190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.447408 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:27.450070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.450029 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:27.549468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.549164 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:27.605988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.584725 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv"] Nov 02 21:37:27.660949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.660921 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:37:27.720995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.720524 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:27.743110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.742582 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:37:27.746117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.746093 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:37:27.856513 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:27.856482 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s"] Nov 02 21:37:28.164244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.163474 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:37:28.231018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.228181 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:37:28.233229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.233192 4441 generic.go:356] "Generic (PLEG): container finished" podID="6a3865cb-8635-4023-bbe9-632c2095371a" containerID="30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81" exitCode=0 Nov 02 21:37:28.233381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.233332 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" event={"ID":"6a3865cb-8635-4023-bbe9-632c2095371a","Type":"ContainerDied","Data":"30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81"} Nov 02 21:37:28.233455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.233388 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" event={"ID":"6a3865cb-8635-4023-bbe9-632c2095371a","Type":"ContainerDied","Data":"080d434d39f9c0d5dc354a9838453e65089ad7aee6ff97c451a619780283b551"} Nov 02 21:37:28.233455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.233417 4441 scope.go:122] "RemoveContainer" containerID="30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81" Nov 02 21:37:28.233612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.233593 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55" Nov 02 21:37:28.242094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.242052 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" event={"ID":"3715402d-8e6d-4348-ae90-e3708cf1960e","Type":"ContainerStarted","Data":"68468c12c56be2b3a1afb493b45a2af7d2a608da0d4b07b3e25c77218e0d0489"} Nov 02 21:37:28.245426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.245403 4441 generic.go:356] "Generic (PLEG): container finished" podID="820029e7-2538-44c4-bd43-a3ad4bae01b3" containerID="9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a" exitCode=0 Nov 02 21:37:28.253027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.246469 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" Nov 02 21:37:28.253422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.253395 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" event={"ID":"820029e7-2538-44c4-bd43-a3ad4bae01b3","Type":"ContainerDied","Data":"9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a"} Nov 02 21:37:28.259085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.253582 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f" event={"ID":"820029e7-2538-44c4-bd43-a3ad4bae01b3","Type":"ContainerDied","Data":"16cca3030cb9f6001c8010530e609f470ee265256df5652b58398876dcb953a9"} Nov 02 21:37:28.280566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.280539 4441 scope.go:122] "RemoveContainer" containerID="30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81" Nov 02 21:37:28.281426 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:37:28.281272 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81\": container with ID starting with 30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81 not found: ID does not exist" containerID="30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81" Nov 02 21:37:28.281426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.281313 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81"} err="failed to get container status \"30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81\": rpc error: code = NotFound desc = could not find container \"30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81\": container with ID starting with 30a6b8726e5f2640ba0baf43cb25198fac93c084c9a9bf61c5bd7d430356ee81 not found: ID does not exist" Nov 02 21:37:28.281426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.281337 4441 scope.go:122] "RemoveContainer" containerID="9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a" Nov 02 21:37:28.318354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.318322 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:37:28.318481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.318399 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:28.335952 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.335879 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820029e7-2538-44c4-bd43-a3ad4bae01b3-kube-api-access-dd4kv\" (UniqueName: \"kubernetes.io/projected/820029e7-2538-44c4-bd43-a3ad4bae01b3-kube-api-access-dd4kv\") pod \"820029e7-2538-44c4-bd43-a3ad4bae01b3\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " Nov 02 21:37:28.336238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.336215 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3865cb-8635-4023-bbe9-632c2095371a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3865cb-8635-4023-bbe9-632c2095371a-plugins-config-volume\") pod \"6a3865cb-8635-4023-bbe9-632c2095371a\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " Nov 02 21:37:28.336514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.336496 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3865cb-8635-4023-bbe9-632c2095371a-kube-api-access-bwzhs\" (UniqueName: \"kubernetes.io/projected/6a3865cb-8635-4023-bbe9-632c2095371a-kube-api-access-bwzhs\") pod \"6a3865cb-8635-4023-bbe9-632c2095371a\" (UID: \"6a3865cb-8635-4023-bbe9-632c2095371a\") " Nov 02 21:37:28.337990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.337785 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820029e7-2538-44c4-bd43-a3ad4bae01b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820029e7-2538-44c4-bd43-a3ad4bae01b3-plugins-config-volume\") pod \"820029e7-2538-44c4-bd43-a3ad4bae01b3\" (UID: \"820029e7-2538-44c4-bd43-a3ad4bae01b3\") " Nov 02 21:37:28.346545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.346520 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820029e7-2538-44c4-bd43-a3ad4bae01b3-plugins-config-volume" pod "820029e7-2538-44c4-bd43-a3ad4bae01b3" (UID: "820029e7-2538-44c4-bd43-a3ad4bae01b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:28.347440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.347419 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3865cb-8635-4023-bbe9-632c2095371a-plugins-config-volume" pod "6a3865cb-8635-4023-bbe9-632c2095371a" (UID: "6a3865cb-8635-4023-bbe9-632c2095371a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:28.403004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.400175 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3865cb-8635-4023-bbe9-632c2095371a-kube-api-access-bwzhs" pod "6a3865cb-8635-4023-bbe9-632c2095371a" (UID: "6a3865cb-8635-4023-bbe9-632c2095371a"). InnerVolumeSpecName "kube-api-access-bwzhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:28.403004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.400274 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820029e7-2538-44c4-bd43-a3ad4bae01b3-kube-api-access-dd4kv" pod "820029e7-2538-44c4-bd43-a3ad4bae01b3" (UID: "820029e7-2538-44c4-bd43-a3ad4bae01b3"). InnerVolumeSpecName "kube-api-access-dd4kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:28.403004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.400397 4441 scope.go:122] "RemoveContainer" containerID="9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a" Nov 02 21:37:28.404134 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:37:28.404102 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a\": container with ID starting with 9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a not found: ID does not exist" containerID="9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a" Nov 02 21:37:28.404311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.404283 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a"} err="failed to get container status \"9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a\": rpc error: code = NotFound desc = could not find container \"9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a\": container with ID starting with 9e5020380dd97cf9cdeb7e345f45a9d860ab0a70e11a6b888a37b01f7c3c372a not found: ID does not exist" Nov 02 21:37:28.481428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.480265 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwzhs\" (UniqueName: \"kubernetes.io/projected/6a3865cb-8635-4023-bbe9-632c2095371a-kube-api-access-bwzhs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:28.481428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.480303 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820029e7-2538-44c4-bd43-a3ad4bae01b3-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:28.481428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.480320 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd4kv\" (UniqueName: \"kubernetes.io/projected/820029e7-2538-44c4-bd43-a3ad4bae01b3-kube-api-access-dd4kv\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:28.481428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.480336 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3865cb-8635-4023-bbe9-632c2095371a-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:28.789824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.783175 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:28.986037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:28.965333 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:37:29.033015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.029224 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55"] Nov 02 21:37:29.051538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.051507 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c39c4123-78b4dd8875-hdg55"] Nov 02 21:37:29.163882 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.163651 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:29.264650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.264531 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" event={"ID":"3715402d-8e6d-4348-ae90-e3708cf1960e","Type":"ContainerStarted","Data":"920d19ca5f9bd2767307b5ab5fcffa009efd79afbc7723d568869023b555461f"} Nov 02 21:37:29.287989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.276678 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f"] Nov 02 21:37:29.289736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.289691 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c39c4123-64c6db69f9-77z5f"] Nov 02 21:37:29.375381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.374388 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" podStartSLOduration=4.374368445 podStartE2EDuration="4.374368445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:29.372415868 +0000 UTC m=+1478.928044205" watchObservedRunningTime="2026-11-02 21:37:29.374368445 +0000 UTC m=+1478.929996782" Nov 02 21:37:29.981025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:29.980986 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:30.276248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:30.276152 4441 generic.go:356] "Generic (PLEG): container finished" podID="3715402d-8e6d-4348-ae90-e3708cf1960e" containerID="920d19ca5f9bd2767307b5ab5fcffa009efd79afbc7723d568869023b555461f" exitCode=0 Nov 02 21:37:30.276620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:30.276244 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" event={"ID":"3715402d-8e6d-4348-ae90-e3708cf1960e","Type":"ContainerDied","Data":"920d19ca5f9bd2767307b5ab5fcffa009efd79afbc7723d568869023b555461f"} Nov 02 21:37:30.589267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:30.589234 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3865cb-8635-4023-bbe9-632c2095371a" path="/var/lib/kubelet/pods/6a3865cb-8635-4023-bbe9-632c2095371a/volumes" Nov 02 21:37:30.589889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:30.589840 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820029e7-2538-44c4-bd43-a3ad4bae01b3" path="/var/lib/kubelet/pods/820029e7-2538-44c4-bd43-a3ad4bae01b3/volumes" Nov 02 21:37:31.510906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.510870 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:31.606307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.606272 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-openshift-ca\") pod \"3715402d-8e6d-4348-ae90-e3708cf1960e\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " Nov 02 21:37:31.606486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.606356 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-xks-ca\") pod \"3715402d-8e6d-4348-ae90-e3708cf1960e\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " Nov 02 21:37:31.606486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.606397 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3715402d-8e6d-4348-ae90-e3708cf1960e-kube-api-access-gt5kn\" (UniqueName: \"kubernetes.io/projected/3715402d-8e6d-4348-ae90-e3708cf1960e-kube-api-access-gt5kn\") pod \"3715402d-8e6d-4348-ae90-e3708cf1960e\" (UID: \"3715402d-8e6d-4348-ae90-e3708cf1960e\") " Nov 02 21:37:31.606752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.606723 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-openshift-ca" pod "3715402d-8e6d-4348-ae90-e3708cf1960e" (UID: "3715402d-8e6d-4348-ae90-e3708cf1960e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:31.607189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.607152 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:31.608671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.608644 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-xks-ca" pod "3715402d-8e6d-4348-ae90-e3708cf1960e" (UID: "3715402d-8e6d-4348-ae90-e3708cf1960e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:31.609183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.609164 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3715402d-8e6d-4348-ae90-e3708cf1960e-kube-api-access-gt5kn" pod "3715402d-8e6d-4348-ae90-e3708cf1960e" (UID: "3715402d-8e6d-4348-ae90-e3708cf1960e"). InnerVolumeSpecName "kube-api-access-gt5kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:31.709226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.709190 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3715402d-8e6d-4348-ae90-e3708cf1960e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:31.709226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:31.709228 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5kn\" (UniqueName: \"kubernetes.io/projected/3715402d-8e6d-4348-ae90-e3708cf1960e-kube-api-access-gt5kn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:32.309406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:32.309364 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" event={"ID":"3715402d-8e6d-4348-ae90-e3708cf1960e","Type":"ContainerDied","Data":"68468c12c56be2b3a1afb493b45a2af7d2a608da0d4b07b3e25c77218e0d0489"} Nov 02 21:37:32.309406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:32.309386 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s" Nov 02 21:37:32.309672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:32.309418 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68468c12c56be2b3a1afb493b45a2af7d2a608da0d4b07b3e25c77218e0d0489" Nov 02 21:37:32.586083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:32.585976 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" podUID="facab52c-2e15-4b8e-bdba-99df0b3703c4" containerName="payload-pre-processing" containerID="cri-o://c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f" gracePeriod=25 Nov 02 21:37:32.888442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:32.888412 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:37:33.031049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.031013 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/facab52c-2e15-4b8e-bdba-99df0b3703c4-kube-api-access-5dgdx\" (UniqueName: \"kubernetes.io/projected/facab52c-2e15-4b8e-bdba-99df0b3703c4-kube-api-access-5dgdx\") pod \"facab52c-2e15-4b8e-bdba-99df0b3703c4\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " Nov 02 21:37:33.031394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.031376 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/facab52c-2e15-4b8e-bdba-99df0b3703c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facab52c-2e15-4b8e-bdba-99df0b3703c4-plugins-config-volume\") pod \"facab52c-2e15-4b8e-bdba-99df0b3703c4\" (UID: \"facab52c-2e15-4b8e-bdba-99df0b3703c4\") " Nov 02 21:37:33.032222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.032198 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facab52c-2e15-4b8e-bdba-99df0b3703c4-plugins-config-volume" pod "facab52c-2e15-4b8e-bdba-99df0b3703c4" (UID: "facab52c-2e15-4b8e-bdba-99df0b3703c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:33.033214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.033189 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facab52c-2e15-4b8e-bdba-99df0b3703c4-kube-api-access-5dgdx" pod "facab52c-2e15-4b8e-bdba-99df0b3703c4" (UID: "facab52c-2e15-4b8e-bdba-99df0b3703c4"). InnerVolumeSpecName "kube-api-access-5dgdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:33.133484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.133406 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facab52c-2e15-4b8e-bdba-99df0b3703c4-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:33.133484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.133436 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dgdx\" (UniqueName: \"kubernetes.io/projected/facab52c-2e15-4b8e-bdba-99df0b3703c4-kube-api-access-5dgdx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:33.322932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.322899 4441 generic.go:356] "Generic (PLEG): container finished" podID="facab52c-2e15-4b8e-bdba-99df0b3703c4" containerID="c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f" exitCode=0 Nov 02 21:37:33.323419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.322946 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" Nov 02 21:37:33.323419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.322994 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" event={"ID":"facab52c-2e15-4b8e-bdba-99df0b3703c4","Type":"ContainerDied","Data":"c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f"} Nov 02 21:37:33.323419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.323051 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv" event={"ID":"facab52c-2e15-4b8e-bdba-99df0b3703c4","Type":"ContainerDied","Data":"54440c32d0aa101eae2247e72267c4c9171fd2e0617d08083f5a01619d598da8"} Nov 02 21:37:33.323419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.323078 4441 scope.go:122] "RemoveContainer" containerID="c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f" Nov 02 21:37:33.401526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.401495 4441 scope.go:122] "RemoveContainer" containerID="c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f" Nov 02 21:37:33.401887 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:37:33.401863 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f\": container with ID starting with c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f not found: ID does not exist" containerID="c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f" Nov 02 21:37:33.401962 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.401895 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f"} err="failed to get container status \"c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f\": rpc error: code = NotFound desc = could not find container \"c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f\": container with ID starting with c60a9bc02bbde51bb6094e2cbdaa0dc23fceeb358be7ae00c414d5b839696f3f not found: ID does not exist" Nov 02 21:37:33.504997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.504900 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv"] Nov 02 21:37:33.510961 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.510932 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ccb9fb8-856588d575-hbzhv"] Nov 02 21:37:33.705252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:33.705166 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:34.334058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:34.334022 4441 generic.go:356] "Generic (PLEG): container finished" podID="85a8a72e-7779-4051-ad2e-5b1669788505" containerID="c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7" exitCode=0 Nov 02 21:37:34.334263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:34.334095 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" event={"ID":"85a8a72e-7779-4051-ad2e-5b1669788505","Type":"ContainerDied","Data":"c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7"} Nov 02 21:37:34.334579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:34.334557 4441 scope.go:122] "RemoveContainer" containerID="c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7" Nov 02 21:37:34.336159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:34.336133 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:34.336241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:34.336197 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:34.336241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:34.336231 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:34.591074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:34.590991 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="facab52c-2e15-4b8e-bdba-99df0b3703c4" path="/var/lib/kubelet/pods/facab52c-2e15-4b8e-bdba-99df0b3703c4/volumes" Nov 02 21:37:35.347813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:35.347779 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" event={"ID":"85a8a72e-7779-4051-ad2e-5b1669788505","Type":"ContainerStarted","Data":"074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2"} Nov 02 21:37:35.737443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:35.737360 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:37:35.738348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:35.738315 4441 patch_prober.go:29] interesting pod/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:37:35.738496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:35.738364 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 21:37:36.136614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:36.136572 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:36.717155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:36.717127 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m"] Nov 02 21:37:36.738073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:36.738045 4441 patch_prober.go:29] interesting pod/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:37:36.738241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:36.738093 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 21:37:37.075993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:37.067423 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:37.194023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:37.193845 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:37.741000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:37.740366 4441 patch_prober.go:29] interesting pod/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:37:37.741000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:37.740445 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 21:37:38.687993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:38.687632 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:38.741006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:38.737508 4441 patch_prober.go:29] interesting pod/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:37:38.741006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:38.737566 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 21:37:39.118572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:39.118544 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:37:39.737759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:39.737730 4441 patch_prober.go:29] interesting pod/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:37:39.737928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:39.737786 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 21:37:40.617145 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:37:40.617109 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:37:40.814300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:40.814263 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:37:40.816082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:40.816055 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:37:41.717463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:41.717422 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" podUID="8c009ac0-b1cc-48ec-ba41-9792bd563285" containerName="payload-processing" containerID="cri-o://3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf" gracePeriod=25 Nov 02 21:37:42.014386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.014356 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:37:42.062918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.062888 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c009ac0-b1cc-48ec-ba41-9792bd563285-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c009ac0-b1cc-48ec-ba41-9792bd563285-plugins-config-volume\") pod \"8c009ac0-b1cc-48ec-ba41-9792bd563285\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " Nov 02 21:37:42.063077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.062940 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c009ac0-b1cc-48ec-ba41-9792bd563285-kube-api-access-xbk57\" (UniqueName: \"kubernetes.io/projected/8c009ac0-b1cc-48ec-ba41-9792bd563285-kube-api-access-xbk57\") pod \"8c009ac0-b1cc-48ec-ba41-9792bd563285\" (UID: \"8c009ac0-b1cc-48ec-ba41-9792bd563285\") " Nov 02 21:37:42.063241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.063220 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c009ac0-b1cc-48ec-ba41-9792bd563285-plugins-config-volume" pod "8c009ac0-b1cc-48ec-ba41-9792bd563285" (UID: "8c009ac0-b1cc-48ec-ba41-9792bd563285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:42.063595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.063575 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c009ac0-b1cc-48ec-ba41-9792bd563285-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:42.065057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.065037 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c009ac0-b1cc-48ec-ba41-9792bd563285-kube-api-access-xbk57" pod "8c009ac0-b1cc-48ec-ba41-9792bd563285" (UID: "8c009ac0-b1cc-48ec-ba41-9792bd563285"). InnerVolumeSpecName "kube-api-access-xbk57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:42.165595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.165565 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbk57\" (UniqueName: \"kubernetes.io/projected/8c009ac0-b1cc-48ec-ba41-9792bd563285-kube-api-access-xbk57\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:42.467936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.467892 4441 generic.go:356] "Generic (PLEG): container finished" podID="8c009ac0-b1cc-48ec-ba41-9792bd563285" containerID="3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf" exitCode=0 Nov 02 21:37:42.467936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.467936 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" Nov 02 21:37:42.468281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.468006 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" event={"ID":"8c009ac0-b1cc-48ec-ba41-9792bd563285","Type":"ContainerDied","Data":"3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf"} Nov 02 21:37:42.468281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.468068 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m" event={"ID":"8c009ac0-b1cc-48ec-ba41-9792bd563285","Type":"ContainerDied","Data":"c18d7a59d27f2b8561afeb95c0b9574db90bef16faa7e10cbe564246b3f42666"} Nov 02 21:37:42.468281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.468099 4441 scope.go:122] "RemoveContainer" containerID="3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf" Nov 02 21:37:42.485849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.485829 4441 scope.go:122] "RemoveContainer" containerID="3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf" Nov 02 21:37:42.486184 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:37:42.486159 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf\": container with ID starting with 3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf not found: ID does not exist" containerID="3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf" Nov 02 21:37:42.486289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.486193 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf"} err="failed to get container status \"3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf\": rpc error: code = NotFound desc = could not find container \"3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf\": container with ID starting with 3b687b7d216be9661f705de5d6931ccb87a1d67b2054fafb94e9592cffd2cecf not found: ID does not exist" Nov 02 21:37:42.608807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.608776 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295"] Nov 02 21:37:42.609447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609433 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6a3865cb-8635-4023-bbe9-632c2095371a" containerName="payload-processing" Nov 02 21:37:42.609508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609449 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3865cb-8635-4023-bbe9-632c2095371a" containerName="payload-processing" Nov 02 21:37:42.609508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609466 4441 cpu_manager.go:414] "containerMap: removing container" podUID="820029e7-2538-44c4-bd43-a3ad4bae01b3" containerName="payload-pre-processing" Nov 02 21:37:42.609508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609472 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820029e7-2538-44c4-bd43-a3ad4bae01b3" containerName="payload-pre-processing" Nov 02 21:37:42.609508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609487 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3715402d-8e6d-4348-ae90-e3708cf1960e" containerName="revoke-keys" Nov 02 21:37:42.609508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609492 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3715402d-8e6d-4348-ae90-e3708cf1960e" containerName="revoke-keys" Nov 02 21:37:42.609508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609500 4441 cpu_manager.go:414] "containerMap: removing container" podUID="facab52c-2e15-4b8e-bdba-99df0b3703c4" containerName="payload-pre-processing" Nov 02 21:37:42.609508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609505 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="facab52c-2e15-4b8e-bdba-99df0b3703c4" containerName="payload-pre-processing" Nov 02 21:37:42.609738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609520 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8c009ac0-b1cc-48ec-ba41-9792bd563285" containerName="payload-processing" Nov 02 21:37:42.609738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609525 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c009ac0-b1cc-48ec-ba41-9792bd563285" containerName="payload-processing" Nov 02 21:37:42.609738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609629 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3865cb-8635-4023-bbe9-632c2095371a" containerName="payload-processing" Nov 02 21:37:42.609738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609639 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="820029e7-2538-44c4-bd43-a3ad4bae01b3" containerName="payload-pre-processing" Nov 02 21:37:42.609738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609647 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3715402d-8e6d-4348-ae90-e3708cf1960e" containerName="revoke-keys" Nov 02 21:37:42.609738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609654 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="facab52c-2e15-4b8e-bdba-99df0b3703c4" containerName="payload-pre-processing" Nov 02 21:37:42.609738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.609661 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c009ac0-b1cc-48ec-ba41-9792bd563285" containerName="payload-processing" Nov 02 21:37:42.613156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.613137 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.631640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.631612 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m"] Nov 02 21:37:42.638441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.638416 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4dff0429-cc8778797-59x9m"] Nov 02 21:37:42.672756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.672728 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzg9\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-kube-api-access-4pzg9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.672890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.672785 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.672890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.672863 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.673097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.673068 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.673199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.673184 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.673284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.673264 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.673426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.673407 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.673469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.673446 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.673518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.673478 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc99d8fb-274c-434d-8918-235d56fb20b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.710950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.710908 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295"] Nov 02 21:37:42.774993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.774887 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc99d8fb-274c-434d-8918-235d56fb20b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.774993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.774954 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzg9\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-kube-api-access-4pzg9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775050 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775084 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775232 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775349 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775411 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775558 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775585 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775612 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775693 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc99d8fb-274c-434d-8918-235d56fb20b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.775751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775692 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.776053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775942 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.776053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.775981 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.777930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.777911 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.778258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.778237 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.828920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.828884 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.829406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.829372 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzg9\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-kube-api-access-4pzg9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw295\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:42.926807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:42.926774 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:43.378174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.378148 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295"] Nov 02 21:37:43.380394 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:37:43.380366 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc99d8fb_274c_434d_8918_235d56fb20b7.slice/crio-590ba63ced125a14ea99a43146f2ced7014e41ba6405ecb2fbaea3967909718a WatchSource:0}: Error finding container 590ba63ced125a14ea99a43146f2ced7014e41ba6405ecb2fbaea3967909718a: Status 404 returned error can't find the container with id 590ba63ced125a14ea99a43146f2ced7014e41ba6405ecb2fbaea3967909718a Nov 02 21:37:43.382945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.382912 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:43.383026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.383012 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:43.383065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.383047 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:37:43.403923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.403895 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:43.485030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.484999 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" event={"ID":"bc99d8fb-274c-434d-8918-235d56fb20b7","Type":"ContainerStarted","Data":"7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071"} Nov 02 21:37:43.485180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.485060 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" event={"ID":"bc99d8fb-274c-434d-8918-235d56fb20b7","Type":"ContainerStarted","Data":"590ba63ced125a14ea99a43146f2ced7014e41ba6405ecb2fbaea3967909718a"} Nov 02 21:37:43.582167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.582076 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:43.600568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.600505 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podStartSLOduration=1.600476541 podStartE2EDuration="1.600476541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:43.594549459 +0000 UTC m=+1493.150177806" watchObservedRunningTime="2026-11-02 21:37:43.600476541 +0000 UTC m=+1493.156104879" Nov 02 21:37:43.927144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:43.927098 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:44.323588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:44.323567 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:44.494010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:44.490512 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:44.605045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:44.604942 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c009ac0-b1cc-48ec-ba41-9792bd563285" path="/var/lib/kubelet/pods/8c009ac0-b1cc-48ec-ba41-9792bd563285/volumes" Nov 02 21:37:44.928382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:44.928283 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 21:37:44.928382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:44.928358 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 21:37:44.932538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:44.932514 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:45.927161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:45.927125 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 21:37:45.927368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:45.927180 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 21:37:46.928103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:46.928067 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 21:37:46.928516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:46.928124 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 21:37:47.927096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:47.927062 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 21:37:47.927351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:47.927112 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 21:37:48.062934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:48.062892 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:48.928166 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:48.928129 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 21:37:48.928345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:48.928189 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 21:37:48.993991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:48.993468 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:49.927387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:49.927350 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 21:37:49.927885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:49.927409 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 21:37:50.884462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.883856 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:37:50.886521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.886487 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:37:50.894581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.894556 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:37:50.896186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.896156 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:37:50.911780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.911758 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:37:50.914774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.914751 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:37:50.927755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.927729 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 02 21:37:50.928064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:50.927792 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 02 21:37:51.173568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:51.173477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:52.256760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:52.256723 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:52.257506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:52.257484 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:52.258240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:52.258224 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:37:52.558455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:52.558420 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:57.048680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:57.048641 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx"] Nov 02 21:37:57.094384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:57.094354 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c42232-5xlpx"] Nov 02 21:37:57.096945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:57.096919 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:58.592884 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:58.592835 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85da4cf-900f-4647-a962-4140606a4a0b" path="/var/lib/kubelet/pods/b85da4cf-900f-4647-a962-4140606a4a0b/volumes" Nov 02 21:37:59.832221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:59.832065 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:37:59.832221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:37:59.832129 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:38:01.099364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:01.099325 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:02.284142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:02.284110 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:02.663999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:02.663927 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:02.974357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:02.974275 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:03.130264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:03.130208 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:03.345276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:03.345239 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7"] Nov 02 21:38:03.395704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:03.395671 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c39c4123-547v7"] Nov 02 21:38:03.437440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:03.437392 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:04.063925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.063891 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:04.184042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.184010 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:04.357488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.357404 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:04.487341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.487312 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:04.645864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.638849 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4393120d-4ac6-4690-b325-a235aaa40df3" path="/var/lib/kubelet/pods/4393120d-4ac6-4690-b325-a235aaa40df3/volumes" Nov 02 21:38:04.749992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.747029 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx"] Nov 02 21:38:04.777117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.755868 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ccb9fb8-ln8vx"] Nov 02 21:38:04.777117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:04.759462 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:05.102009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:05.101949 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:05.348129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:05.348099 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:05.493312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:05.493224 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:06.589197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:06.589150 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd19779-7474-42c0-a057-5a1d7e6d2746" path="/var/lib/kubelet/pods/9bd19779-7474-42c0-a057-5a1d7e6d2746/volumes" Nov 02 21:38:07.329690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:07.329649 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:07.538353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:07.538299 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:09.221032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:09.220979 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:09.367441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:09.367404 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:09.507201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:09.507119 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:10.619594 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:38:10.619548 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:38:11.399582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:11.399534 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:12.188829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.188793 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:12.446225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.446151 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:12.580040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.575992 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:12.703499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.703418 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:12.799217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.799178 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s"] Nov 02 21:38:12.826932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.826899 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4dff0429-9qb9s"] Nov 02 21:38:12.836736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.836704 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh"] Nov 02 21:38:12.837001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.836959 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" containerID="cri-o://086035d08106eb970e559613b826083c7018f892c51ea5316d095ddf862ba1a6" gracePeriod=30 Nov 02 21:38:12.841912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:12.841794 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:13.147077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:13.147044 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:13.356715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:13.356639 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:13.494729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:13.494652 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:13.710081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:13.710047 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:13.846015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:13.845986 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:13.973674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:13.973643 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:14.604733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:14.604691 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3715402d-8e6d-4348-ae90-e3708cf1960e" path="/var/lib/kubelet/pods/3715402d-8e6d-4348-ae90-e3708cf1960e/volumes" Nov 02 21:38:14.706586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:14.706549 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:16.748229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:16.748190 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:17.059450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:17.059419 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6"] Nov 02 21:38:17.106052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:17.106021 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedc90f3-xrtf6"] Nov 02 21:38:17.186486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:17.186451 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:17.359899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:17.359814 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:18.229543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.229455 4441 generic.go:356] "Generic (PLEG): container finished" podID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerID="086035d08106eb970e559613b826083c7018f892c51ea5316d095ddf862ba1a6" exitCode=0 Nov 02 21:38:18.230516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.230482 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" event={"ID":"dc184427-21af-49cc-b2fd-07f5426aedaf","Type":"ContainerDied","Data":"086035d08106eb970e559613b826083c7018f892c51ea5316d095ddf862ba1a6"} Nov 02 21:38:18.230647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.230600 4441 scope.go:122] "RemoveContainer" containerID="99acbefa13ac9e1aa413d2dfa618f790376b09692a9dd4d806be42af73850761" Nov 02 21:38:18.269680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.269587 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:38:18.335611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.335579 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc184427-21af-49cc-b2fd-07f5426aedaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc184427-21af-49cc-b2fd-07f5426aedaf-istiod-ca-cert\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.335838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.335695 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-data\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.335838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.335747 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-podinfo\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.335838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.335785 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-token\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-token\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.336121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.335871 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-socket\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.336121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.335922 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-envoy\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.336121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.335950 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-certs\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.336121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336044 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-kube-api-access-gf5hg\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-kube-api-access-gf5hg\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.336121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336059 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-data" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:18.336121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336085 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc184427-21af-49cc-b2fd-07f5426aedaf-istiod-ca-cert" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:18.336121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336099 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-credential-socket\") pod \"dc184427-21af-49cc-b2fd-07f5426aedaf\" (UID: \"dc184427-21af-49cc-b2fd-07f5426aedaf\") " Nov 02 21:38:18.336454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336291 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-certs" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:18.336454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336425 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-credential-socket" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:18.336610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336589 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-socket" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:18.336980 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336929 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.336980 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.336953 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.337137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.337000 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.337137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.337016 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc184427-21af-49cc-b2fd-07f5426aedaf-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.337137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.337030 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.338373 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.338340 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-kube-api-access-gf5hg" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "kube-api-access-gf5hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:18.338476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.338435 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-podinfo" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:38:18.338809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.338791 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-envoy" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:18.338866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.338814 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-token" pod "dc184427-21af-49cc-b2fd-07f5426aedaf" (UID: "dc184427-21af-49cc-b2fd-07f5426aedaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:18.438601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.438565 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5hg\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-kube-api-access-gf5hg\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.438601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.438595 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.438601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.438606 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.438888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.438615 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc184427-21af-49cc-b2fd-07f5426aedaf-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:18.588206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:18.588164 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48dcd6d8-5266-4d2f-8b51-2577fbc431a5" path="/var/lib/kubelet/pods/48dcd6d8-5266-4d2f-8b51-2577fbc431a5/volumes" Nov 02 21:38:19.240516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:19.240489 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" Nov 02 21:38:19.240936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:19.240524 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh" event={"ID":"dc184427-21af-49cc-b2fd-07f5426aedaf","Type":"ContainerDied","Data":"6b89efe2d2002274745f96836f5c659767c045f3a63095dbca775fa1362df58c"} Nov 02 21:38:19.240936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:19.240583 4441 scope.go:122] "RemoveContainer" containerID="086035d08106eb970e559613b826083c7018f892c51ea5316d095ddf862ba1a6" Nov 02 21:38:19.475892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:19.475857 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh"] Nov 02 21:38:19.484532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:19.484500 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4dff0429-openshift-default-559487f9c7-22gzh"] Nov 02 21:38:20.587571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:20.587534 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" path="/var/lib/kubelet/pods/dc184427-21af-49cc-b2fd-07f5426aedaf/volumes" Nov 02 21:38:21.046823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:21.046744 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m"] Nov 02 21:38:21.093929 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:21.093889 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-633c334c-tsf2m"] Nov 02 21:38:21.758840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:21.758782 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:22.404275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.404240 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l"] Nov 02 21:38:22.404938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.404923 4441 cpu_manager.go:414] "containerMap: removing container" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" Nov 02 21:38:22.405026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.404939 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" Nov 02 21:38:22.405026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.404959 4441 cpu_manager.go:414] "containerMap: removing container" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" Nov 02 21:38:22.405026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.404976 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" Nov 02 21:38:22.405138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.405128 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" Nov 02 21:38:22.405173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.405141 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc184427-21af-49cc-b2fd-07f5426aedaf" containerName="istio-proxy" Nov 02 21:38:22.408867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.408851 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.501031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.500998 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.501214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.501066 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.501299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.501270 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c721ccdd-2315-40fb-9a76-956febbcb546-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.501410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.501383 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgzp\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-kube-api-access-qmgzp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.501572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.501547 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.501710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.501617 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.501710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.501646 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.506029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.506005 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l"] Nov 02 21:38:22.585623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.585587 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222530ae-1b3b-4226-85eb-ae7f8bd6f3f8" path="/var/lib/kubelet/pods/222530ae-1b3b-4226-85eb-ae7f8bd6f3f8/volumes" Nov 02 21:38:22.603027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.602991 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.603189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.603077 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.603189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.603105 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.603306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.603247 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.603306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.603288 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.603440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.603426 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c721ccdd-2315-40fb-9a76-956febbcb546-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.603494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.603479 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgzp\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-kube-api-access-qmgzp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.603917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.603894 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.605688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.605657 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c721ccdd-2315-40fb-9a76-956febbcb546-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.605865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.605844 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.605943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.605927 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.606098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.606076 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.657580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.657507 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgzp\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-kube-api-access-qmgzp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.657580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.657514 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n2q6l\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:22.722916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:22.722879 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:23.168239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:23.168207 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l"] Nov 02 21:38:23.211693 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:38:23.211663 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc721ccdd_2315_40fb_9a76_956febbcb546.slice/crio-98a3e8a29f431d2267fe6b9f29a998c5d555805f34a256b1720d837c5373efd6 WatchSource:0}: Error finding container 98a3e8a29f431d2267fe6b9f29a998c5d555805f34a256b1720d837c5373efd6: Status 404 returned error can't find the container with id 98a3e8a29f431d2267fe6b9f29a998c5d555805f34a256b1720d837c5373efd6 Nov 02 21:38:23.214277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:23.214247 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:38:23.214343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:23.214310 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:38:23.281436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:23.281408 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" event={"ID":"c721ccdd-2315-40fb-9a76-956febbcb546","Type":"ContainerStarted","Data":"98a3e8a29f431d2267fe6b9f29a998c5d555805f34a256b1720d837c5373efd6"} Nov 02 21:38:23.281543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:23.281471 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:23.396322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:23.396267 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" podStartSLOduration=1.396253309 podStartE2EDuration="1.396253309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:23.392318413 +0000 UTC m=+1532.947946751" watchObservedRunningTime="2026-11-02 21:38:23.396253309 +0000 UTC m=+1532.951881645" Nov 02 21:38:24.292718 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:24.292676 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" event={"ID":"c721ccdd-2315-40fb-9a76-956febbcb546","Type":"ContainerStarted","Data":"1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067"} Nov 02 21:38:24.295032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:24.295006 4441 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-n2q6l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:38:24.295168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:24.295057 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" podUID="c721ccdd-2315-40fb-9a76-956febbcb546" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:38:25.302824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:25.302792 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:38:26.267893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:26.267853 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:30.779307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:30.779272 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:31.046881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:31.046788 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t"] Nov 02 21:38:31.094103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:31.094064 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ddfffc-q5p9t"] Nov 02 21:38:32.046825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:32.046789 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd"] Nov 02 21:38:32.097605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:32.097570 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0cdac9-52knd"] Nov 02 21:38:32.589667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:32.589627 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8d1813-c9b3-4811-b133-d3c2952822fb" path="/var/lib/kubelet/pods/2b8d1813-c9b3-4811-b133-d3c2952822fb/volumes" Nov 02 21:38:32.590190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:32.590170 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7ecc8e-f35b-4d79-ba0f-e62e6bded104" path="/var/lib/kubelet/pods/7b7ecc8e-f35b-4d79-ba0f-e62e6bded104/volumes" Nov 02 21:38:34.058145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:34.058107 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:34.234699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:34.234659 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:35.276139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:35.276102 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:39.800670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:39.800635 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:40.617137 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:38:40.617096 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:38:41.045597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:41.045507 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:44.290427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:44.290389 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:47.307962 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:47.307905 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:48.788071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:48.788022 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:49.050532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:49.050454 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr"] Nov 02 21:38:49.100781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:49.100743 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25467252-jg7lr"] Nov 02 21:38:49.145445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:49.145401 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:49.268379 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:49.268349 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:50.487963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:50.487928 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:50.602260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:50.602227 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23e46eb-0250-4933-9de6-74cd41b3b94a" path="/var/lib/kubelet/pods/a23e46eb-0250-4933-9de6-74cd41b3b94a/volumes" Nov 02 21:38:53.234035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:53.233951 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:53.380730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:53.380692 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:53.508586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:53.508478 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:55.053585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:55.053549 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png"] Nov 02 21:38:55.059890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:55.059858 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7ed3418-v7png"] Nov 02 21:38:56.586336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:56.586291 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5809dc-15a3-4789-9fb8-9b6e398d57cf" path="/var/lib/kubelet/pods/7d5809dc-15a3-4789-9fb8-9b6e398d57cf/volumes" Nov 02 21:38:56.814549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:56.814502 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:57.944877 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:57.944848 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:58.106866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:58.106794 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:58.231524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:38:58.231442 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:00.145800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:00.145767 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:00.311004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:00.305579 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:00.596755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:00.596723 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:02.978682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:02.978581 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:04.674094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:04.674057 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b"] Nov 02 21:39:04.674651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:04.674274 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" containerID="cri-o://8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5" gracePeriod=30 Nov 02 21:39:04.719858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:04.719826 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc"] Nov 02 21:39:04.720094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:04.720071 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" containerID="cri-o://e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00" gracePeriod=30 Nov 02 21:39:05.048934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:05.048842 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp"] Nov 02 21:39:05.054779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:05.054751 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58f9b4f-tjgrp"] Nov 02 21:39:06.125509 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:06.125473 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:06.266790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:06.266763 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:06.586496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:06.586464 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc759d7-bd00-4e59-b3cf-3486fba4e78b" path="/var/lib/kubelet/pods/afc759d7-bd00-4e59-b3cf-3486fba4e78b/volumes" Nov 02 21:39:07.049199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:07.049123 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk"] Nov 02 21:39:07.096064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:07.096034 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66013425-874jk"] Nov 02 21:39:07.590510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:07.590480 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:08.586811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:08.586775 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8417dfbf-35cf-40f9-b6e0-3bbb1454297e" path="/var/lib/kubelet/pods/8417dfbf-35cf-40f9-b6e0-3bbb1454297e/volumes" Nov 02 21:39:10.057443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.057400 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:39:10.109869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.109839 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:39:10.249655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.249617 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792b7c65-d723-4944-9465-6691ce7a0876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792b7c65-d723-4944-9465-6691ce7a0876-istiod-ca-cert\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.249812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.249693 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-certs\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.249812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.249738 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-data\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.249812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.249803 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-envoy\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.250052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.249859 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-data\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.250052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.249889 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-kube-api-access-4p85s\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-kube-api-access-4p85s\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.250052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.249923 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-kube-api-access-mkddb\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-kube-api-access-mkddb\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.250052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250022 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-certs\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.250266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250077 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792b7c65-d723-4944-9465-6691ce7a0876-istiod-ca-cert" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:10.250266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250118 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-socket\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.250266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250158 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istiod-ca-cert\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.250266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250191 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-socket\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.250266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250204 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-data" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.250535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250287 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-credential-socket\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.250535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250357 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-token\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.250535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250394 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/792b7c65-d723-4944-9465-6691ce7a0876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792b7c65-d723-4944-9465-6691ce7a0876-istio-podinfo\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.250535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250467 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-credential-socket\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.250752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250586 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-envoy\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.250752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250614 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-istio-token\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-istio-token\") pod \"792b7c65-d723-4944-9465-6691ce7a0876\" (UID: \"792b7c65-d723-4944-9465-6691ce7a0876\") " Nov 02 21:39:10.250752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250644 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-podinfo\") pod \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\" (UID: \"1ed97436-88ee-4c2f-9418-91ec4b3e28e9\") " Nov 02 21:39:10.251488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.251459 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792b7c65-d723-4944-9465-6691ce7a0876-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.251610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.251492 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250610 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-certs" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250697 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-certs" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.250893 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-data" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.251040 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-socket" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.251118 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-credential-socket" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.251296 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-socket" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.251827 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istiod-ca-cert" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:10.252744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.252110 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-credential-socket" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.253758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.253619 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-envoy" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.253758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.253720 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-kube-api-access-4p85s" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "kube-api-access-4p85s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:10.253758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.253754 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-envoy" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:10.254019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.253774 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-token" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:10.254019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.253808 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-kube-api-access-mkddb" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "kube-api-access-mkddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:10.254783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.254754 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-podinfo" pod "1ed97436-88ee-4c2f-9418-91ec4b3e28e9" (UID: "1ed97436-88ee-4c2f-9418-91ec4b3e28e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:39:10.255028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.255002 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-istio-token" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:10.255323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.255305 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/792b7c65-d723-4944-9465-6691ce7a0876-istio-podinfo" pod "792b7c65-d723-4944-9465-6691ce7a0876" (UID: "792b7c65-d723-4944-9465-6691ce7a0876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:39:10.353190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353156 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353184 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792b7c65-d723-4944-9465-6691ce7a0876-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353194 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353202 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353213 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353222 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353231 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353240 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353247 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353257 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p85s\" (UniqueName: \"kubernetes.io/projected/792b7c65-d723-4944-9465-6691ce7a0876-kube-api-access-4p85s\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353266 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkddb\" (UniqueName: \"kubernetes.io/projected/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-kube-api-access-mkddb\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353274 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353283 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353293 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed97436-88ee-4c2f-9418-91ec4b3e28e9-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353301 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.353426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.353310 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792b7c65-d723-4944-9465-6691ce7a0876-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:10.618096 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:39:10.618015 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:39:10.772453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.772415 4441 generic.go:356] "Generic (PLEG): container finished" podID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerID="8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5" exitCode=0 Nov 02 21:39:10.772627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.772475 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" Nov 02 21:39:10.772627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.772503 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" event={"ID":"1ed97436-88ee-4c2f-9418-91ec4b3e28e9","Type":"ContainerDied","Data":"8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5"} Nov 02 21:39:10.772627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.772563 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b" event={"ID":"1ed97436-88ee-4c2f-9418-91ec4b3e28e9","Type":"ContainerDied","Data":"b2c05f85c8cf88aafd20865b5b050607b95c207f48ecfae38a38b0453c37d42e"} Nov 02 21:39:10.772627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.772600 4441 scope.go:122] "RemoveContainer" containerID="8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5" Nov 02 21:39:10.774548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.774510 4441 generic.go:356] "Generic (PLEG): container finished" podID="792b7c65-d723-4944-9465-6691ce7a0876" containerID="e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00" exitCode=0 Nov 02 21:39:10.775557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.774786 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" event={"ID":"792b7c65-d723-4944-9465-6691ce7a0876","Type":"ContainerDied","Data":"e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00"} Nov 02 21:39:10.775557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.774833 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" event={"ID":"792b7c65-d723-4944-9465-6691ce7a0876","Type":"ContainerDied","Data":"dc6b44b49c8dfa7c7cf6c30e7ad6726b0bff0fbfc94f60fa5aecce2d70800bd3"} Nov 02 21:39:10.775557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.774902 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc" Nov 02 21:39:10.787776 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.787753 4441 scope.go:122] "RemoveContainer" containerID="8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5" Nov 02 21:39:10.788128 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:39:10.788101 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5\": container with ID starting with 8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5 not found: ID does not exist" containerID="8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5" Nov 02 21:39:10.788239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.788137 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5"} err="failed to get container status \"8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5\": rpc error: code = NotFound desc = could not find container \"8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5\": container with ID starting with 8f2b30c4e91040751bec799ebafbf94c6b5bdfedbf5e09f53d6c9ea658a7e9d5 not found: ID does not exist" Nov 02 21:39:10.788239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.788163 4441 scope.go:122] "RemoveContainer" containerID="e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00" Nov 02 21:39:10.801554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.801533 4441 scope.go:122] "RemoveContainer" containerID="e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00" Nov 02 21:39:10.801843 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:39:10.801822 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00\": container with ID starting with e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00 not found: ID does not exist" containerID="e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00" Nov 02 21:39:10.801954 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.801849 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00"} err="failed to get container status \"e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00\": rpc error: code = NotFound desc = could not find container \"e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00\": container with ID starting with e081cc03dca9db8025c93c302d19cf4bebe5d078fcb26ab1d33f09fe6b2d1d00 not found: ID does not exist" Nov 02 21:39:10.933506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.933440 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b"] Nov 02 21:39:10.983517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:10.983484 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wv79b"] Nov 02 21:39:11.038400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:11.038370 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc"] Nov 02 21:39:11.044119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:11.044097 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-wjhwc"] Nov 02 21:39:11.049261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:11.049242 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp"] Nov 02 21:39:11.054074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:11.054054 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e670b198-v52rp"] Nov 02 21:39:12.103511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:12.103464 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:12.586978 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:12.586936 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" path="/var/lib/kubelet/pods/1ed97436-88ee-4c2f-9418-91ec4b3e28e9/volumes" Nov 02 21:39:12.587432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:12.587416 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f25a0d-bad9-42c3-bc04-f92933273134" path="/var/lib/kubelet/pods/75f25a0d-bad9-42c3-bc04-f92933273134/volumes" Nov 02 21:39:12.587851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:12.587837 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792b7c65-d723-4944-9465-6691ce7a0876" path="/var/lib/kubelet/pods/792b7c65-d723-4944-9465-6691ce7a0876/volumes" Nov 02 21:39:15.725490 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.725457 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76"] Nov 02 21:39:15.726164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.726148 4441 cpu_manager.go:414] "containerMap: removing container" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" Nov 02 21:39:15.726164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.726165 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" Nov 02 21:39:15.726260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.726176 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" Nov 02 21:39:15.726260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.726181 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" Nov 02 21:39:15.726326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.726295 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="792b7c65-d723-4944-9465-6691ce7a0876" containerName="istio-proxy" Nov 02 21:39:15.726326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.726305 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed97436-88ee-4c2f-9418-91ec4b3e28e9" containerName="istio-proxy" Nov 02 21:39:15.730872 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.730855 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.784723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.784692 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76"] Nov 02 21:39:15.822597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.822558 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:39:15.822766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.822607 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:39:15.822766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.822629 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:39:15.827256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.827236 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:39:15.831163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.831138 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.831291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.831210 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.831432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.831406 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45wj\" (UniqueName: \"kubernetes.io/projected/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-kube-api-access-j45wj\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.934126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.934090 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.934304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.934219 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.934363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.934346 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45wj\" (UniqueName: \"kubernetes.io/projected/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-kube-api-access-j45wj\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.936057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.936027 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.946265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.946233 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:15.988487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:15.988427 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45wj\" (UniqueName: \"kubernetes.io/projected/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-kube-api-access-j45wj\") pod \"maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:16.045424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:16.045397 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:16.046804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:16.046775 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42"] Nov 02 21:39:16.052587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:16.052555 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5218240-scx42"] Nov 02 21:39:16.441556 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:16.441531 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76"] Nov 02 21:39:16.444798 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:39:16.444769 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b62aa69_fcab_4af1_bf6a_6c5395d4a18d.slice/crio-7f0c8b6bfd5ac8cb9c5d1ba816ed1261ce4b2c35ee0734217c9ae6774a214a4f WatchSource:0}: Error finding container 7f0c8b6bfd5ac8cb9c5d1ba816ed1261ce4b2c35ee0734217c9ae6774a214a4f: Status 404 returned error can't find the container with id 7f0c8b6bfd5ac8cb9c5d1ba816ed1261ce4b2c35ee0734217c9ae6774a214a4f Nov 02 21:39:16.590049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:16.590008 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4387fa0b-7c13-4ff2-b852-bc7bca78bac9" path="/var/lib/kubelet/pods/4387fa0b-7c13-4ff2-b852-bc7bca78bac9/volumes" Nov 02 21:39:16.740522 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:16.740442 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:16.849950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:16.849917 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" event={"ID":"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d","Type":"ContainerStarted","Data":"7f0c8b6bfd5ac8cb9c5d1ba816ed1261ce4b2c35ee0734217c9ae6774a214a4f"} Nov 02 21:39:17.860752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:17.860720 4441 generic.go:356] "Generic (PLEG): container finished" podID="7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" containerID="707b09bab7b293ef1f49d35858fb1aaa48c0f22f73c9ffa2465242679bd0cccb" exitCode=0 Nov 02 21:39:17.861250 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:17.860759 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" event={"ID":"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d","Type":"ContainerDied","Data":"707b09bab7b293ef1f49d35858fb1aaa48c0f22f73c9ffa2465242679bd0cccb"} Nov 02 21:39:19.057892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.057864 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:19.181395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.181307 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-kube-api-access-j45wj\" (UniqueName: \"kubernetes.io/projected/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-kube-api-access-j45wj\") pod \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " Nov 02 21:39:19.183036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.182961 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-openshift-ca\") pod \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " Nov 02 21:39:19.183216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.183196 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-xks-ca\") pod \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\" (UID: \"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d\") " Nov 02 21:39:19.189026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.188957 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-xks-ca" pod "7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" (UID: "7b62aa69-fcab-4af1-bf6a-6c5395d4a18d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:39:19.189359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.189334 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-openshift-ca" pod "7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" (UID: "7b62aa69-fcab-4af1-bf6a-6c5395d4a18d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:19.264882 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.264843 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-kube-api-access-j45wj" pod "7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" (UID: "7b62aa69-fcab-4af1-bf6a-6c5395d4a18d"). InnerVolumeSpecName "kube-api-access-j45wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:19.297199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.286538 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:19.297199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.286581 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j45wj\" (UniqueName: \"kubernetes.io/projected/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-kube-api-access-j45wj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:19.297199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.286598 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:19.932144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.932102 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" event={"ID":"7b62aa69-fcab-4af1-bf6a-6c5395d4a18d","Type":"ContainerDied","Data":"7f0c8b6bfd5ac8cb9c5d1ba816ed1261ce4b2c35ee0734217c9ae6774a214a4f"} Nov 02 21:39:19.932813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.932778 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f0c8b6bfd5ac8cb9c5d1ba816ed1261ce4b2c35ee0734217c9ae6774a214a4f" Nov 02 21:39:19.947152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:19.933149 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76" Nov 02 21:39:22.263250 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:22.263220 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:24.214856 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:24.214822 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:24.913194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:24.913159 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:25.324256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:25.324218 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:27.041471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:27.041437 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:27.264839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:27.264803 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr"] Nov 02 21:39:27.615256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:27.615221 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:27.746155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:27.746118 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:27.889041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:27.877854 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:28.053692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:28.053650 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf"] Nov 02 21:39:28.060608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:28.060579 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848cbd60-g5sdf"] Nov 02 21:39:28.579434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:28.579402 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:28.585728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:28.585701 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db76fec-1d0f-448a-aecd-b0f00648cef8" path="/var/lib/kubelet/pods/0db76fec-1d0f-448a-aecd-b0f00648cef8/volumes" Nov 02 21:39:29.047031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:29.046941 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk"] Nov 02 21:39:29.075847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:29.075810 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01dce43d-bwnnk"] Nov 02 21:39:30.509208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:30.509172 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:30.587375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:30.587345 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a467613-ece7-441a-a190-73a008fd332b" path="/var/lib/kubelet/pods/3a467613-ece7-441a-a190-73a008fd332b/volumes" Nov 02 21:39:30.726960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:30.726928 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:30.856013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:30.855946 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:32.265351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.265310 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" podUID="c5519e76-3d13-4b6e-b751-2d0f09c2244a" containerName="payload-processing" containerID="cri-o://9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5" gracePeriod=25 Nov 02 21:39:32.574765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.574739 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:39:32.682940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.682904 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5519e76-3d13-4b6e-b751-2d0f09c2244a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5519e76-3d13-4b6e-b751-2d0f09c2244a-plugins-config-volume\") pod \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " Nov 02 21:39:32.683308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.683275 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5519e76-3d13-4b6e-b751-2d0f09c2244a-kube-api-access-4vsnt\" (UniqueName: \"kubernetes.io/projected/c5519e76-3d13-4b6e-b751-2d0f09c2244a-kube-api-access-4vsnt\") pod \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\" (UID: \"c5519e76-3d13-4b6e-b751-2d0f09c2244a\") " Nov 02 21:39:32.683850 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.683823 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5519e76-3d13-4b6e-b751-2d0f09c2244a-plugins-config-volume" pod "c5519e76-3d13-4b6e-b751-2d0f09c2244a" (UID: "c5519e76-3d13-4b6e-b751-2d0f09c2244a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:32.684278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.684254 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5519e76-3d13-4b6e-b751-2d0f09c2244a-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:32.685873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.685837 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5519e76-3d13-4b6e-b751-2d0f09c2244a-kube-api-access-4vsnt" pod "c5519e76-3d13-4b6e-b751-2d0f09c2244a" (UID: "c5519e76-3d13-4b6e-b751-2d0f09c2244a"). InnerVolumeSpecName "kube-api-access-4vsnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:32.786139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:32.786061 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsnt\" (UniqueName: \"kubernetes.io/projected/c5519e76-3d13-4b6e-b751-2d0f09c2244a-kube-api-access-4vsnt\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:33.093087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.093055 4441 generic.go:356] "Generic (PLEG): container finished" podID="c5519e76-3d13-4b6e-b751-2d0f09c2244a" containerID="9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5" exitCode=0 Nov 02 21:39:33.093269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.093127 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" Nov 02 21:39:33.093269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.093145 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" event={"ID":"c5519e76-3d13-4b6e-b751-2d0f09c2244a","Type":"ContainerDied","Data":"9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5"} Nov 02 21:39:33.093269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.093198 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr" event={"ID":"c5519e76-3d13-4b6e-b751-2d0f09c2244a","Type":"ContainerDied","Data":"2eca53f29c3ac9da4bcd59950c14655b3b2497e7ac7adb35bff43caef5585dbe"} Nov 02 21:39:33.093269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.093222 4441 scope.go:122] "RemoveContainer" containerID="9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5" Nov 02 21:39:33.105590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.105570 4441 scope.go:122] "RemoveContainer" containerID="9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5" Nov 02 21:39:33.105858 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:39:33.105840 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5\": container with ID starting with 9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5 not found: ID does not exist" containerID="9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5" Nov 02 21:39:33.105929 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.105869 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5"} err="failed to get container status \"9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5\": rpc error: code = NotFound desc = could not find container \"9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5\": container with ID starting with 9a7666a5139717aa53e184d6507f87bea9808c7737ed4edbd981e6bcb9ae46c5 not found: ID does not exist" Nov 02 21:39:33.179106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.179075 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:33.253637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.253607 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr"] Nov 02 21:39:33.300200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.300173 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7cd2d37-5cb657cf9c-cbzrr"] Nov 02 21:39:33.402649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.402570 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:33.526533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.526503 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:33.672949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.672870 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:33.844234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:33.844204 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:34.055783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:34.054931 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554"] Nov 02 21:39:34.059715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:34.059673 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:34.065630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:34.065602 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416d259f-st554"] Nov 02 21:39:34.586214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:34.586182 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5689eb08-4fdf-4f80-a4a0-d31419df2c73" path="/var/lib/kubelet/pods/5689eb08-4fdf-4f80-a4a0-d31419df2c73/volumes" Nov 02 21:39:34.586729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:34.586714 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5519e76-3d13-4b6e-b751-2d0f09c2244a" path="/var/lib/kubelet/pods/c5519e76-3d13-4b6e-b751-2d0f09c2244a/volumes" Nov 02 21:39:36.046849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:36.046804 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2"] Nov 02 21:39:36.093634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:36.093603 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df5ca53d-9wzh2"] Nov 02 21:39:36.586832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:36.586800 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51" path="/var/lib/kubelet/pods/cfd1bdfa-cf68-4bd4-830d-6f6fd6b1de51/volumes" Nov 02 21:39:38.281478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.281439 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:38.500088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.488483 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:38.701492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.701460 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn"] Nov 02 21:39:38.702243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.702221 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" containerName="revoke-keys" Nov 02 21:39:38.702243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.702243 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" containerName="revoke-keys" Nov 02 21:39:38.702429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.702286 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c5519e76-3d13-4b6e-b751-2d0f09c2244a" containerName="payload-processing" Nov 02 21:39:38.702429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.702294 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5519e76-3d13-4b6e-b751-2d0f09c2244a" containerName="payload-processing" Nov 02 21:39:38.702530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.702461 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5519e76-3d13-4b6e-b751-2d0f09c2244a" containerName="payload-processing" Nov 02 21:39:38.702530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.702487 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" containerName="revoke-keys" Nov 02 21:39:38.708768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.708744 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:38.756700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.756669 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn"] Nov 02 21:39:38.797272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.796673 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f4a242-dockercfg-6csgj\"" Nov 02 21:39:38.797272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.796813 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f4a242\"" Nov 02 21:39:38.816230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.814425 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7fc\" (UniqueName: \"kubernetes.io/projected/dcb6c070-74c2-4dd1-9173-28e6020c3142-kube-api-access-td7fc\") pod \"payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:38.816230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:38.814536 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb6c070-74c2-4dd1-9173-28e6020c3142-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:39.024039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.021343 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7fc\" (UniqueName: \"kubernetes.io/projected/dcb6c070-74c2-4dd1-9173-28e6020c3142-kube-api-access-td7fc\") pod \"payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:39.024039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.021448 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb6c070-74c2-4dd1-9173-28e6020c3142-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:39.024039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.022740 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb6c070-74c2-4dd1-9173-28e6020c3142-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:39.050017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.036066 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6"] Nov 02 21:39:39.077240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.070389 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.091332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.091305 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6"] Nov 02 21:39:39.091913 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.091889 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7fc\" (UniqueName: \"kubernetes.io/projected/dcb6c070-74c2-4dd1-9173-28e6020c3142-kube-api-access-td7fc\") pod \"payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:39.124532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.124172 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbffaf2e-9895-426c-85db-11c650e7306d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.124532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.124360 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hctjl\" (UniqueName: \"kubernetes.io/projected/fbffaf2e-9895-426c-85db-11c650e7306d-kube-api-access-hctjl\") pod \"payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.225700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.225665 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbffaf2e-9895-426c-85db-11c650e7306d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.225871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.225799 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hctjl\" (UniqueName: \"kubernetes.io/projected/fbffaf2e-9895-426c-85db-11c650e7306d-kube-api-access-hctjl\") pod \"payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.226493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.226468 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbffaf2e-9895-426c-85db-11c650e7306d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.279509 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.279437 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hctjl\" (UniqueName: \"kubernetes.io/projected/fbffaf2e-9895-426c-85db-11c650e7306d-kube-api-access-hctjl\") pod \"payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.330338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.330307 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:39.438608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.438580 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:39.493027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.492708 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:39.748194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.748168 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn"] Nov 02 21:39:39.751362 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:39:39.751337 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb6c070_74c2_4dd1_9173_28e6020c3142.slice/crio-33908d073f6891413d37d39beeb55e1101c44edf9f5f58dad596c5416a683311 WatchSource:0}: Error finding container 33908d073f6891413d37d39beeb55e1101c44edf9f5f58dad596c5416a683311: Status 404 returned error can't find the container with id 33908d073f6891413d37d39beeb55e1101c44edf9f5f58dad596c5416a683311 Nov 02 21:39:39.860538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:39.860431 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6"] Nov 02 21:39:39.863342 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:39:39.863317 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbffaf2e_9895_426c_85db_11c650e7306d.slice/crio-1303971963b0ca0dda452322c83dda3bbbde6d2a6122aa6d0160be024c249235 WatchSource:0}: Error finding container 1303971963b0ca0dda452322c83dda3bbbde6d2a6122aa6d0160be024c249235: Status 404 returned error can't find the container with id 1303971963b0ca0dda452322c83dda3bbbde6d2a6122aa6d0160be024c249235 Nov 02 21:39:40.171596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.171553 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" event={"ID":"dcb6c070-74c2-4dd1-9173-28e6020c3142","Type":"ContainerStarted","Data":"b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9"} Nov 02 21:39:40.171800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.171660 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:40.171800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.171684 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" event={"ID":"dcb6c070-74c2-4dd1-9173-28e6020c3142","Type":"ContainerStarted","Data":"33908d073f6891413d37d39beeb55e1101c44edf9f5f58dad596c5416a683311"} Nov 02 21:39:40.173811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.173784 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" event={"ID":"fbffaf2e-9895-426c-85db-11c650e7306d","Type":"ContainerStarted","Data":"d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79"} Nov 02 21:39:40.173946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.173827 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" event={"ID":"fbffaf2e-9895-426c-85db-11c650e7306d","Type":"ContainerStarted","Data":"1303971963b0ca0dda452322c83dda3bbbde6d2a6122aa6d0160be024c249235"} Nov 02 21:39:40.173946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.173915 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:40.284738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.284683 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" podStartSLOduration=2.284670377 podStartE2EDuration="2.284670377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:40.279379501 +0000 UTC m=+1609.835007839" watchObservedRunningTime="2026-11-02 21:39:40.284670377 +0000 UTC m=+1609.840298715" Nov 02 21:39:40.395722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:40.395657 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" podStartSLOduration=2.395638069 podStartE2EDuration="2.395638069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:40.386001846 +0000 UTC m=+1609.941630178" watchObservedRunningTime="2026-11-02 21:39:40.395638069 +0000 UTC m=+1609.951266407" Nov 02 21:39:40.624253 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:39:40.624213 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:39:42.046294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:42.046258 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv"] Nov 02 21:39:42.093597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:42.093559 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c0d14d3-r2nsv"] Nov 02 21:39:42.585804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:42.585772 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ebfb50-c8eb-4632-b8ff-57d42415e459" path="/var/lib/kubelet/pods/e4ebfb50-c8eb-4632-b8ff-57d42415e459/volumes" Nov 02 21:39:44.048655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:44.048612 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg"] Nov 02 21:39:44.094413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:44.094383 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4841349-dhtsg"] Nov 02 21:39:44.267305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:44.267270 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:44.585657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:44.585580 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c4c94d-9929-4db3-b7b2-3c1031cb75ef" path="/var/lib/kubelet/pods/b8c4c94d-9929-4db3-b7b2-3c1031cb75ef/volumes" Nov 02 21:39:46.046087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:46.046055 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm"] Nov 02 21:39:46.093481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:46.093448 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765"] Nov 02 21:39:46.098852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:46.098826 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1850e5f8-pmftm"] Nov 02 21:39:46.103207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:46.103185 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3a3d2f-fs765"] Nov 02 21:39:46.592864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:46.592830 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a6be75-ac93-474c-b41a-8234bb782416" path="/var/lib/kubelet/pods/c9a6be75-ac93-474c-b41a-8234bb782416/volumes" Nov 02 21:39:46.593378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:46.593364 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e483842a-c8b7-4513-9e5f-12a8e6c8cafd" path="/var/lib/kubelet/pods/e483842a-c8b7-4513-9e5f-12a8e6c8cafd/volumes" Nov 02 21:39:47.045232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:47.045159 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt"] Nov 02 21:39:47.091720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:47.091691 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e3d1fa-hcpmt"] Nov 02 21:39:48.586220 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:48.586189 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b242b9f-c0d6-49d2-9d78-a0efbc37385e" path="/var/lib/kubelet/pods/1b242b9f-c0d6-49d2-9d78-a0efbc37385e/volumes" Nov 02 21:39:48.806504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:48.806470 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:49.491244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:49.491212 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:51.045192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.045159 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk"] Nov 02 21:39:51.095050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.095009 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569c8718-dkzwk"] Nov 02 21:39:51.189268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.189239 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 21:39:51.189450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.189327 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 21:39:51.615017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.614986 4441 scope.go:122] "RemoveContainer" containerID="7d0d22f215f133403451db1acce65cecd5a6915a6cf4bc48c68dfcea96e0b941" Nov 02 21:39:51.630685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.630658 4441 scope.go:122] "RemoveContainer" containerID="f2a8f914ac74b15d90548f21e00c7d8253de928dd7dccba61c79e73b61265309" Nov 02 21:39:51.640761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.640731 4441 scope.go:122] "RemoveContainer" containerID="0a61bfb31f564c0495b26be84da914acfb770c76b91cc93a27b799f38279253b" Nov 02 21:39:51.650590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.650568 4441 scope.go:122] "RemoveContainer" containerID="5e8eab0fe28a9bb5316cc6d8a2c29be26210d1c9b51da49bc3eb15ccfea83482" Nov 02 21:39:51.659820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.659799 4441 scope.go:122] "RemoveContainer" containerID="6fad5b304d22266db8190f5be372f279ef1098b45eb5f02fa82e10b70bcc7491" Nov 02 21:39:51.669580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.669559 4441 scope.go:122] "RemoveContainer" containerID="19a6bc11df362fc8de44c43774aa9fa6d34ec1c595f98b798905e2546323a312" Nov 02 21:39:51.681775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:51.681753 4441 scope.go:122] "RemoveContainer" containerID="5fb86449d54347e1b5ea765d30e1c86f85488d7bcb92873237179fc4faf6013f" Nov 02 21:39:52.587986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:52.587947 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e519b02e-deed-4936-b1e5-545668bafb67" path="/var/lib/kubelet/pods/e519b02e-deed-4936-b1e5-545668bafb67/volumes" Nov 02 21:39:53.600593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:53.600552 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:56.055681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:56.055644 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq"] Nov 02 21:39:56.060512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:56.060484 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e9ab4d-pbmxq"] Nov 02 21:39:56.585389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:56.585358 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e86e58-ea15-4ea1-8b7a-3f2929d0a318" path="/var/lib/kubelet/pods/96e86e58-ea15-4ea1-8b7a-3f2929d0a318/volumes" Nov 02 21:39:57.532071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.532037 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z"] Nov 02 21:39:57.538836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.538814 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.621439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.621406 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:39:57.621439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.621414 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:39:57.626713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.626665 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:39:57.626713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.626690 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:39:57.629992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.629948 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z"] Nov 02 21:39:57.652607 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.652580 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.652801 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.652773 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6mw\" (UniqueName: \"kubernetes.io/projected/db93f7dc-666c-460e-9b4d-672eee6e79c7-kube-api-access-xc6mw\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.653018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.652989 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.786048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.765156 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6mw\" (UniqueName: \"kubernetes.io/projected/db93f7dc-666c-460e-9b4d-672eee6e79c7-kube-api-access-xc6mw\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.786048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.770502 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.786048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.770862 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.787231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.787196 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.814021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.809523 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.830732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.830689 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6mw\" (UniqueName: \"kubernetes.io/projected/db93f7dc-666c-460e-9b4d-672eee6e79c7-kube-api-access-xc6mw\") pod \"maas-api-revoke-keys-e2e-ait-5935b530-xtw2z\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:57.860992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:57.860246 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:39:58.286862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:58.286829 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z"] Nov 02 21:39:58.291227 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:39:58.291204 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb93f7dc_666c_460e_9b4d_672eee6e79c7.slice/crio-faf7f908ee57a8e815a033e932a5af1c482d8c89673964ed74efb0ce006c7edf WatchSource:0}: Error finding container faf7f908ee57a8e815a033e932a5af1c482d8c89673964ed74efb0ce006c7edf: Status 404 returned error can't find the container with id faf7f908ee57a8e815a033e932a5af1c482d8c89673964ed74efb0ce006c7edf Nov 02 21:39:58.358387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:58.358353 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" event={"ID":"db93f7dc-666c-460e-9b4d-672eee6e79c7","Type":"ContainerStarted","Data":"faf7f908ee57a8e815a033e932a5af1c482d8c89673964ed74efb0ce006c7edf"} Nov 02 21:39:58.739874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:58.739838 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:59.369221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:59.369192 4441 generic.go:356] "Generic (PLEG): container finished" podID="db93f7dc-666c-460e-9b4d-672eee6e79c7" containerID="bc6b4c69c82a52274f9271c0e7f9d45f02f986d948108032142be1e24a44304d" exitCode=0 Nov 02 21:39:59.369365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:39:59.369289 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" event={"ID":"db93f7dc-666c-460e-9b4d-672eee6e79c7","Type":"ContainerDied","Data":"bc6b4c69c82a52274f9271c0e7f9d45f02f986d948108032142be1e24a44304d"} Nov 02 21:40:00.553220 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.553195 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:40:00.725461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.725372 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db93f7dc-666c-460e-9b4d-672eee6e79c7-kube-api-access-xc6mw\" (UniqueName: \"kubernetes.io/projected/db93f7dc-666c-460e-9b4d-672eee6e79c7-kube-api-access-xc6mw\") pod \"db93f7dc-666c-460e-9b4d-672eee6e79c7\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " Nov 02 21:40:00.725616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.725468 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-openshift-ca\") pod \"db93f7dc-666c-460e-9b4d-672eee6e79c7\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " Nov 02 21:40:00.725616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.725538 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-xks-ca\") pod \"db93f7dc-666c-460e-9b4d-672eee6e79c7\" (UID: \"db93f7dc-666c-460e-9b4d-672eee6e79c7\") " Nov 02 21:40:00.726266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.726235 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-openshift-ca" pod "db93f7dc-666c-460e-9b4d-672eee6e79c7" (UID: "db93f7dc-666c-460e-9b4d-672eee6e79c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:00.726924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.726899 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:00.727831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.727803 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-xks-ca" pod "db93f7dc-666c-460e-9b4d-672eee6e79c7" (UID: "db93f7dc-666c-460e-9b4d-672eee6e79c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:00.728466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.728431 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db93f7dc-666c-460e-9b4d-672eee6e79c7-kube-api-access-xc6mw" pod "db93f7dc-666c-460e-9b4d-672eee6e79c7" (UID: "db93f7dc-666c-460e-9b4d-672eee6e79c7"). InnerVolumeSpecName "kube-api-access-xc6mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:00.828519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.828491 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db93f7dc-666c-460e-9b4d-672eee6e79c7-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:00.828519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:00.828516 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6mw\" (UniqueName: \"kubernetes.io/projected/db93f7dc-666c-460e-9b4d-672eee6e79c7-kube-api-access-xc6mw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:01.389393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:01.389366 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" Nov 02 21:40:01.389597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:01.389406 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z" event={"ID":"db93f7dc-666c-460e-9b4d-672eee6e79c7","Type":"ContainerDied","Data":"faf7f908ee57a8e815a033e932a5af1c482d8c89673964ed74efb0ce006c7edf"} Nov 02 21:40:01.389597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:01.389450 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faf7f908ee57a8e815a033e932a5af1c482d8c89673964ed74efb0ce006c7edf" Nov 02 21:40:02.046124 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:02.046083 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg"] Nov 02 21:40:02.092736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:02.092707 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21292c82-qx5pg"] Nov 02 21:40:02.586641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:02.586603 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f7842a-62b4-4ad8-9556-55a250cfc104" path="/var/lib/kubelet/pods/f1f7842a-62b4-4ad8-9556-55a250cfc104/volumes" Nov 02 21:40:03.044722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:03.044642 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7"] Nov 02 21:40:03.091853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:03.091811 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b07d639-7znh7"] Nov 02 21:40:03.747836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:03.747806 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:04.045264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:04.045180 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp"] Nov 02 21:40:04.092671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:04.092636 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f0864-rcrwp"] Nov 02 21:40:04.588584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:04.588530 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eae43ad-a3e4-4915-9a22-beb654744c1f" path="/var/lib/kubelet/pods/2eae43ad-a3e4-4915-9a22-beb654744c1f/volumes" Nov 02 21:40:04.589050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:04.589033 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a46bb5-e6f6-44af-b056-0221640aa117" path="/var/lib/kubelet/pods/78a46bb5-e6f6-44af-b056-0221640aa117/volumes" Nov 02 21:40:05.240883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.240846 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4"] Nov 02 21:40:05.241543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.241527 4441 cpu_manager.go:414] "containerMap: removing container" podUID="db93f7dc-666c-460e-9b4d-672eee6e79c7" containerName="revoke-keys" Nov 02 21:40:05.241609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.241544 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db93f7dc-666c-460e-9b4d-672eee6e79c7" containerName="revoke-keys" Nov 02 21:40:05.241699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.241689 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="db93f7dc-666c-460e-9b4d-672eee6e79c7" containerName="revoke-keys" Nov 02 21:40:05.246722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.246704 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.298690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.298648 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpb5\" (UniqueName: \"kubernetes.io/projected/d62382b7-3cef-4b3b-b4b9-4d000b69180f-kube-api-access-xzpb5\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.298868 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.298825 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.298936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.298867 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.299260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.299234 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4"] Nov 02 21:40:05.339287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.339256 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:40:05.339506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.339261 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:40:05.361273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.361244 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:40:05.386617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.386580 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:40:05.400495 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.400467 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpb5\" (UniqueName: \"kubernetes.io/projected/d62382b7-3cef-4b3b-b4b9-4d000b69180f-kube-api-access-xzpb5\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.400661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.400599 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.400661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.400636 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.401458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.401429 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.403182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.403159 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.454384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.454355 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpb5\" (UniqueName: \"kubernetes.io/projected/d62382b7-3cef-4b3b-b4b9-4d000b69180f-kube-api-access-xzpb5\") pod \"maas-api-revoke-keys-e2e-ait-e942516f-6qwd4\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.560200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.560172 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:05.955745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:05.955603 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4"] Nov 02 21:40:05.959191 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:40:05.959163 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62382b7_3cef_4b3b_b4b9_4d000b69180f.slice/crio-27c796ff5ef96e974a17b349cfda4cecbad0cd6a08394925bcbec335cefc0b57 WatchSource:0}: Error finding container 27c796ff5ef96e974a17b349cfda4cecbad0cd6a08394925bcbec335cefc0b57: Status 404 returned error can't find the container with id 27c796ff5ef96e974a17b349cfda4cecbad0cd6a08394925bcbec335cefc0b57 Nov 02 21:40:06.441256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:06.441220 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" event={"ID":"d62382b7-3cef-4b3b-b4b9-4d000b69180f","Type":"ContainerStarted","Data":"034bd5f54840006bf94b6dc833933815b66e0cb621967fce48f4dcb1772da218"} Nov 02 21:40:06.441651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:06.441282 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" event={"ID":"d62382b7-3cef-4b3b-b4b9-4d000b69180f","Type":"ContainerStarted","Data":"27c796ff5ef96e974a17b349cfda4cecbad0cd6a08394925bcbec335cefc0b57"} Nov 02 21:40:06.551206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:06.551154 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" podStartSLOduration=1.55114057 podStartE2EDuration="1.55114057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:06.548180957 +0000 UTC m=+1636.103809293" watchObservedRunningTime="2026-11-02 21:40:06.55114057 +0000 UTC m=+1636.106768910" Nov 02 21:40:07.451995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:07.451929 4441 generic.go:356] "Generic (PLEG): container finished" podID="d62382b7-3cef-4b3b-b4b9-4d000b69180f" containerID="034bd5f54840006bf94b6dc833933815b66e0cb621967fce48f4dcb1772da218" exitCode=0 Nov 02 21:40:07.452512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:07.452017 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" event={"ID":"d62382b7-3cef-4b3b-b4b9-4d000b69180f","Type":"ContainerDied","Data":"034bd5f54840006bf94b6dc833933815b66e0cb621967fce48f4dcb1772da218"} Nov 02 21:40:07.858340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:07.858281 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:08.743229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.743204 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:08.802689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.802660 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:08.884643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.884602 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-xks-ca\") pod \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " Nov 02 21:40:08.884826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.884730 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62382b7-3cef-4b3b-b4b9-4d000b69180f-kube-api-access-xzpb5\" (UniqueName: \"kubernetes.io/projected/d62382b7-3cef-4b3b-b4b9-4d000b69180f-kube-api-access-xzpb5\") pod \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " Nov 02 21:40:08.884910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.884892 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-openshift-ca\") pod \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\" (UID: \"d62382b7-3cef-4b3b-b4b9-4d000b69180f\") " Nov 02 21:40:08.886301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.886272 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-openshift-ca" pod "d62382b7-3cef-4b3b-b4b9-4d000b69180f" (UID: "d62382b7-3cef-4b3b-b4b9-4d000b69180f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:08.891156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.891114 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-xks-ca" pod "d62382b7-3cef-4b3b-b4b9-4d000b69180f" (UID: "d62382b7-3cef-4b3b-b4b9-4d000b69180f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:08.891535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.891498 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62382b7-3cef-4b3b-b4b9-4d000b69180f-kube-api-access-xzpb5" pod "d62382b7-3cef-4b3b-b4b9-4d000b69180f" (UID: "d62382b7-3cef-4b3b-b4b9-4d000b69180f"). InnerVolumeSpecName "kube-api-access-xzpb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:08.942832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.942118 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:09.005050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.987639 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:09.005050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.987676 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzpb5\" (UniqueName: \"kubernetes.io/projected/d62382b7-3cef-4b3b-b4b9-4d000b69180f-kube-api-access-xzpb5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:09.005050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:08.987695 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62382b7-3cef-4b3b-b4b9-4d000b69180f-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:09.472730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:09.472699 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" Nov 02 21:40:09.472918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:09.472729 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4" event={"ID":"d62382b7-3cef-4b3b-b4b9-4d000b69180f","Type":"ContainerDied","Data":"27c796ff5ef96e974a17b349cfda4cecbad0cd6a08394925bcbec335cefc0b57"} Nov 02 21:40:09.472918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:09.472784 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27c796ff5ef96e974a17b349cfda4cecbad0cd6a08394925bcbec335cefc0b57" Nov 02 21:40:09.673984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:09.673940 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:09.871302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:09.871266 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:09.992337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:09.992301 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:10.261622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:10.261530 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76"] Nov 02 21:40:10.271075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:10.271041 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7cd2d37-f9k76"] Nov 02 21:40:10.589426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:10.589392 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b62aa69-fcab-4af1-bf6a-6c5395d4a18d" path="/var/lib/kubelet/pods/7b62aa69-fcab-4af1-bf6a-6c5395d4a18d/volumes" Nov 02 21:40:10.607266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:10.607224 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:10.623840 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:40:10.623811 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:40:11.571926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:11.571888 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:12.046075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.045956 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c"] Nov 02 21:40:12.097598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.097521 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2984bd0a-hbd5c"] Nov 02 21:40:12.103761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.103731 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg"] Nov 02 21:40:12.105265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.105236 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d62382b7-3cef-4b3b-b4b9-4d000b69180f" containerName="revoke-keys" Nov 02 21:40:12.105265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.105265 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62382b7-3cef-4b3b-b4b9-4d000b69180f" containerName="revoke-keys" Nov 02 21:40:12.105784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.105508 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62382b7-3cef-4b3b-b4b9-4d000b69180f" containerName="revoke-keys" Nov 02 21:40:12.109979 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.109941 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.166297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.166230 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg"] Nov 02 21:40:12.192592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.192552 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-06366c-openshift-default-dockercfg-vwwf2\"" Nov 02 21:40:12.195309 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.195277 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:12.267015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.266979 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-credential-socket\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267071 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-istio-token\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267267 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367798ae-c8d3-444b-a221-99eec69e544c-istiod-ca-cert\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267343 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-data\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267396 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-certs\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267423 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-socket\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267445 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367798ae-c8d3-444b-a221-99eec69e544c-istio-podinfo\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267545 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-envoy\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.267593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.267588 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxfv\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-kube-api-access-6sxfv\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.368901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.368860 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367798ae-c8d3-444b-a221-99eec69e544c-istiod-ca-cert\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.368933 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-data\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369020 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-certs\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369058 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-socket\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369089 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367798ae-c8d3-444b-a221-99eec69e544c-istio-podinfo\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-envoy\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369185 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxfv\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-kube-api-access-6sxfv\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369396 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-credential-socket\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369486 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-istio-token\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.369730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.369703 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367798ae-c8d3-444b-a221-99eec69e544c-istiod-ca-cert\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.374087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.370363 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-credential-socket\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.374087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.370644 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-socket\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.374087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.370700 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-certs\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.374087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.370869 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-data\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.379256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.379226 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-envoy\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.379744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.379670 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367798ae-c8d3-444b-a221-99eec69e544c-istio-podinfo\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.431104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.430847 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-istio-token\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.431104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.431060 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxfv\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-kube-api-access-6sxfv\") pod \"e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:12.587405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.587369 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaedf9a3-6eae-4435-9a5c-35a36ff16a40" path="/var/lib/kubelet/pods/aaedf9a3-6eae-4435-9a5c-35a36ff16a40/volumes" Nov 02 21:40:12.588668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.588640 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:12.729126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:12.729012 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:13.203576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.203520 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg"] Nov 02 21:40:13.252989 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:40:13.246755 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367798ae_c8d3_444b_a221_99eec69e544c.slice/crio-126453da549dab897e67a0bff33c2e1b442f672fc9c6743735cb7aace6ecf3ca WatchSource:0}: Error finding container 126453da549dab897e67a0bff33c2e1b442f672fc9c6743735cb7aace6ecf3ca: Status 404 returned error can't find the container with id 126453da549dab897e67a0bff33c2e1b442f672fc9c6743735cb7aace6ecf3ca Nov 02 21:40:13.252989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.252640 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:40:13.252989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.252755 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:40:13.252989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.252798 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:40:13.537963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.537864 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" event={"ID":"367798ae-c8d3-444b-a221-99eec69e544c","Type":"ContainerStarted","Data":"da38b0b3c6255f9ce62b3513a7183f12ae0e157eef368e2b9ae737f0a3dc1e60"} Nov 02 21:40:13.537963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.537918 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" event={"ID":"367798ae-c8d3-444b-a221-99eec69e544c","Type":"ContainerStarted","Data":"126453da549dab897e67a0bff33c2e1b442f672fc9c6743735cb7aace6ecf3ca"} Nov 02 21:40:13.651347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.651296 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podStartSLOduration=1.651281264 podStartE2EDuration="1.651281264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:13.64810834 +0000 UTC m=+1643.203736702" watchObservedRunningTime="2026-11-02 21:40:13.651281264 +0000 UTC m=+1643.206909601" Nov 02 21:40:13.729645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:13.729614 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:14.045035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:14.045000 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx"] Nov 02 21:40:14.050019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:14.049994 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f12db8-nw4sx"] Nov 02 21:40:14.585566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:14.585537 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b13166-e7e0-4e41-ab46-ff31e5b49b29" path="/var/lib/kubelet/pods/b9b13166-e7e0-4e41-ab46-ff31e5b49b29/volumes" Nov 02 21:40:14.735762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:14.735734 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:15.046269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:15.046168 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj"] Nov 02 21:40:15.051186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:15.051156 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d4da39-dc6vj"] Nov 02 21:40:15.123655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:15.123624 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:15.556826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:15.556795 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:15.557674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:15.557651 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:40:16.471376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:16.471346 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:16.587247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:16.587219 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fb272b-6df7-4a87-a2d9-9447e929538f" path="/var/lib/kubelet/pods/56fb272b-6df7-4a87-a2d9-9447e929538f/volumes" Nov 02 21:40:16.679512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:16.679481 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:16.834287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:16.834251 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:17.109811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:17.109728 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm"] Nov 02 21:40:17.110034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:17.109933 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" containerID="cri-o://074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2" gracePeriod=30 Nov 02 21:40:17.140528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:17.134517 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:17.525987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:17.525862 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:17.660700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:17.660667 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:17.794235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:17.794132 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:18.036431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:18.036392 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:20.393803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:20.393764 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:21.095442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:21.095404 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:21.309453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:21.309419 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:21.458325 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:21.458223 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:22.051429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.051384 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m"] Nov 02 21:40:22.095685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.095652 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6e3fd4-rgh9m"] Nov 02 21:40:22.414717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.414690 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:40:22.487195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487143 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-credential-socket\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.487673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487301 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-envoy\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.487673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487395 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a8a72e-7779-4051-ad2e-5b1669788505-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a8a72e-7779-4051-ad2e-5b1669788505-istiod-ca-cert\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.487673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487448 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-credential-socket" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:22.487673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487462 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85a8a72e-7779-4051-ad2e-5b1669788505-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a8a72e-7779-4051-ad2e-5b1669788505-istio-podinfo\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.487673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487559 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-kube-api-access-pnxws\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-kube-api-access-pnxws\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.487673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487601 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-socket\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.488049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487698 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-certs\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.488049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487734 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a8a72e-7779-4051-ad2e-5b1669788505-istiod-ca-cert" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:22.488049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487800 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-data\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.488049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.487880 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-istio-token\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-istio-token\") pod \"85a8a72e-7779-4051-ad2e-5b1669788505\" (UID: \"85a8a72e-7779-4051-ad2e-5b1669788505\") " Nov 02 21:40:22.488550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.488525 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-certs" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:22.488733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.488712 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a8a72e-7779-4051-ad2e-5b1669788505-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.488797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.488739 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.488797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.488767 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.488797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.488764 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-socket" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:22.488960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.488932 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-data" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:22.490363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.490282 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-envoy" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:22.490721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.490700 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85a8a72e-7779-4051-ad2e-5b1669788505-istio-podinfo" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:22.490721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.491681 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-istio-token" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:22.493321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.493294 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-kube-api-access-pnxws" pod "85a8a72e-7779-4051-ad2e-5b1669788505" (UID: "85a8a72e-7779-4051-ad2e-5b1669788505"). InnerVolumeSpecName "kube-api-access-pnxws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:22.619253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.619214 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.619253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.619255 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a8a72e-7779-4051-ad2e-5b1669788505-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.619552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.619271 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxws\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-kube-api-access-pnxws\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.619552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.619288 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.619552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.619303 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a8a72e-7779-4051-ad2e-5b1669788505-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.619552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.619317 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a8a72e-7779-4051-ad2e-5b1669788505-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:22.649360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.649258 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfdf6f47-fe9c-4676-90f1-9da71e75c1d8" path="/var/lib/kubelet/pods/cfdf6f47-fe9c-4676-90f1-9da71e75c1d8/volumes" Nov 02 21:40:22.669073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.669037 4441 generic.go:356] "Generic (PLEG): container finished" podID="85a8a72e-7779-4051-ad2e-5b1669788505" containerID="074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2" exitCode=0 Nov 02 21:40:22.669397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.669375 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" event={"ID":"85a8a72e-7779-4051-ad2e-5b1669788505","Type":"ContainerDied","Data":"074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2"} Nov 02 21:40:22.669491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.669431 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" event={"ID":"85a8a72e-7779-4051-ad2e-5b1669788505","Type":"ContainerDied","Data":"f3753253c23c90165ef4ce669ab259c325e6793f4ed36c2514d3347618f83bd8"} Nov 02 21:40:22.669491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.669460 4441 scope.go:122] "RemoveContainer" containerID="074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2" Nov 02 21:40:22.669669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.669644 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm" Nov 02 21:40:22.683675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.683649 4441 scope.go:122] "RemoveContainer" containerID="c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7" Nov 02 21:40:22.695536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.695506 4441 scope.go:122] "RemoveContainer" containerID="074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2" Nov 02 21:40:22.695842 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:40:22.695807 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2\": container with ID starting with 074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2 not found: ID does not exist" containerID="074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2" Nov 02 21:40:22.695918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.695855 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2"} err="failed to get container status \"074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2\": rpc error: code = NotFound desc = could not find container \"074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2\": container with ID starting with 074dd2ac4dbb5c034771919e2e6d9fcf5c63a97c014c1733737c0a3c8af48ac2 not found: ID does not exist" Nov 02 21:40:22.695918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.695879 4441 scope.go:122] "RemoveContainer" containerID="c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7" Nov 02 21:40:22.696192 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:40:22.696172 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7\": container with ID starting with c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7 not found: ID does not exist" containerID="c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7" Nov 02 21:40:22.696256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.696201 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7"} err="failed to get container status \"c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7\": rpc error: code = NotFound desc = could not find container \"c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7\": container with ID starting with c2fb8da407789960315ec044af7fa2b011b55d226f641521b1f1c201ea53aac7 not found: ID does not exist" Nov 02 21:40:22.828605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.828575 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm"] Nov 02 21:40:22.833393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.833368 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c7cd2d37-openshift-default-6458d58469-rkntm"] Nov 02 21:40:22.860337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.860310 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg"] Nov 02 21:40:22.860992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.860963 4441 cpu_manager.go:414] "containerMap: removing container" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" Nov 02 21:40:22.861041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.860994 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" Nov 02 21:40:22.861041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.861003 4441 cpu_manager.go:414] "containerMap: removing container" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" Nov 02 21:40:22.861041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.861008 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" Nov 02 21:40:22.861144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.861135 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" Nov 02 21:40:22.861180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.861147 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" containerName="istio-proxy" Nov 02 21:40:22.866446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.866425 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:22.916052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.915984 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b7107c-openshift-default-dockercfg-5pklh\"" Nov 02 21:40:22.942696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:22.942668 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:23.033993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.033862 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-token\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.033993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.033946 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istiod-ca-cert\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.034246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.034000 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-envoy\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.034246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.034068 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-podinfo\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.034246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.034184 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-socket\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.034408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.034263 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-data\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.034408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.034325 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5wl\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-kube-api-access-ct5wl\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.034408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.034401 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-certs\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.034551 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.034474 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-credential-socket\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.063132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.062912 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg"] Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136028 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-data\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136114 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5wl\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-kube-api-access-ct5wl\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136190 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-certs\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136259 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-credential-socket\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136352 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-token\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136407 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istiod-ca-cert\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136444 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-envoy\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136514 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-podinfo\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.136647 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-socket\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.137410 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-socket\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.137680 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-data\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.138378 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-certs\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.138902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.138612 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-credential-socket\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.139634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.139606 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istiod-ca-cert\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.141227 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.141202 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:23.155168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.154985 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-podinfo\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.155365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.155340 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-envoy\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.197077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.197000 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5wl\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-kube-api-access-ct5wl\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.197493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.197470 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-token\") pod \"e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.484301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.484219 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:23.752379 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.752306 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:23.881036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.881002 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:23.938222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.938191 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg"] Nov 02 21:40:23.941834 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:40:23.941808 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ddcfe4_e1a2_4e98_adee_d30a1297583f.slice/crio-efd0f09841656366894f30dc4a9b0fa22e5d9ba85ee34f9366465388c638afe8 WatchSource:0}: Error finding container efd0f09841656366894f30dc4a9b0fa22e5d9ba85ee34f9366465388c638afe8: Status 404 returned error can't find the container with id efd0f09841656366894f30dc4a9b0fa22e5d9ba85ee34f9366465388c638afe8 Nov 02 21:40:23.944491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.944461 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:40:23.944553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.944526 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:40:23.944595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:23.944555 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:40:24.068340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.068306 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz"] Nov 02 21:40:24.086074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.086047 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84eb51c4-c5tjz"] Nov 02 21:40:24.237017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.236945 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:24.586447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.586414 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc53ef2-3cf2-4417-a110-d76abc393828" path="/var/lib/kubelet/pods/4cc53ef2-3cf2-4417-a110-d76abc393828/volumes" Nov 02 21:40:24.586988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.586959 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a8a72e-7779-4051-ad2e-5b1669788505" path="/var/lib/kubelet/pods/85a8a72e-7779-4051-ad2e-5b1669788505/volumes" Nov 02 21:40:24.692231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.692195 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" event={"ID":"f1ddcfe4-e1a2-4e98-adee-d30a1297583f","Type":"ContainerStarted","Data":"adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd"} Nov 02 21:40:24.692413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.692250 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" event={"ID":"f1ddcfe4-e1a2-4e98-adee-d30a1297583f","Type":"ContainerStarted","Data":"efd0f09841656366894f30dc4a9b0fa22e5d9ba85ee34f9366465388c638afe8"} Nov 02 21:40:24.849424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.849338 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:24.898842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:24.898789 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podStartSLOduration=2.898776428 podStartE2EDuration="2.898776428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:24.849751538 +0000 UTC m=+1654.405379875" watchObservedRunningTime="2026-11-02 21:40:24.898776428 +0000 UTC m=+1654.454404765" Nov 02 21:40:25.076598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.071550 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:25.279457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.274747 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:25.484814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.484781 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:25.491122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.491095 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:25.535579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.535480 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:25.670196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.670160 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:25.704633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.704600 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:25.705963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:25.705940 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:40:27.693901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:27.693871 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:27.863377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:27.863340 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:27.999567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:27.999478 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:30.488721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:30.488686 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:30.626270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:30.626235 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:30.801275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:30.799754 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:30.954428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:30.954297 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:31.088831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:31.088801 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:31.996678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:31.996560 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:32.044754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:32.044720 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x"] Nov 02 21:40:32.091671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:32.091640 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eca6925-zgm6x"] Nov 02 21:40:32.162301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:32.162268 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:32.370426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:32.370382 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:32.589721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:32.589681 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25b3dff-2d67-4683-9d59-5b994ed91792" path="/var/lib/kubelet/pods/d25b3dff-2d67-4683-9d59-5b994ed91792/volumes" Nov 02 21:40:35.924234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:35.924199 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:39.804039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:39.804000 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:39.995990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:39.995512 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:40.617959 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:40:40.617918 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:40:41.095563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:41.095524 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:43.047344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:43.047309 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8"] Nov 02 21:40:43.093107 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:43.093074 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd8d12a-mcvd8"] Nov 02 21:40:44.586401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:44.586313 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252fabe8-441b-42b5-ad3d-0ad9c29815e6" path="/var/lib/kubelet/pods/252fabe8-441b-42b5-ad3d-0ad9c29815e6/volumes" Nov 02 21:40:46.054806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:46.054770 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:49.815857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:49.815823 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:49.956753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:49.956716 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:51.151640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.151594 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:51.546594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.546496 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:51.781656 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.781621 4441 scope.go:122] "RemoveContainer" containerID="000ad8f7ba3267f3e34ef0cdfb102f45774a8666b64f5a86077a5ca9e505afc8" Nov 02 21:40:51.805246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.805207 4441 scope.go:122] "RemoveContainer" containerID="685bee53f2f7b911607f57932cb708c8daa699f06f50d1c0fc1955716e3bc784" Nov 02 21:40:51.818646 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.818625 4441 scope.go:122] "RemoveContainer" containerID="99c13d8a00e0319c3717e808e42c8a7ea83d037777ebb09ce8af7ecab9ac5cb2" Nov 02 21:40:51.830549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.830524 4441 scope.go:122] "RemoveContainer" containerID="f7e4280df6bee81e9b82c73cc9ecabb20345881ab35413aef3c8efcf2584cd34" Nov 02 21:40:51.840394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.840363 4441 scope.go:122] "RemoveContainer" containerID="687c30bf864989560447d87ca97558b237c2f014fb8e7c66b3c6490a7f449027" Nov 02 21:40:51.849922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.849901 4441 scope.go:122] "RemoveContainer" containerID="3c439a2145120fb385146aa3f7c6f53534b78ce6dc3f14a76c73d617d92bef40" Nov 02 21:40:51.859893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.859869 4441 scope.go:122] "RemoveContainer" containerID="eea66cd686759cc4ad61d0a81943e07bc98402330debcaa2131487e4831252ec" Nov 02 21:40:51.873326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.873301 4441 scope.go:122] "RemoveContainer" containerID="9827eaa95cdd6ce5e2ba97c599640a33c3843cea7df1c4430fd6c9c2b000649f" Nov 02 21:40:51.888768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.888745 4441 scope.go:122] "RemoveContainer" containerID="1f31ee32ba2a6fc11bfbac48c1ea21623c1bada006393e1d579c3e97b6aa0987" Nov 02 21:40:51.923860 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.923830 4441 scope.go:122] "RemoveContainer" containerID="895e8c77cb1a975af788b8c3e75e6d59d7185fb72edab104c222f50b2d2ffa1d" Nov 02 21:40:51.935481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.935461 4441 scope.go:122] "RemoveContainer" containerID="0a46a6f67bee898220565b72fbc92599e8ea07c035f508df08420550566e166f" Nov 02 21:40:51.947021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.946962 4441 scope.go:122] "RemoveContainer" containerID="2d569643198c189784a6204d662f6256ac7a237fff8c2d9fdfe8d232bd3e8fe1" Nov 02 21:40:51.954848 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.954827 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:51.957529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.957510 4441 scope.go:122] "RemoveContainer" containerID="e7527f830864d6997f601cf7c237446133ce0ad1cbdd710885790eb921dd885d" Nov 02 21:40:51.967330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:51.967312 4441 scope.go:122] "RemoveContainer" containerID="6ca67c462561bae9bf3835d87258d14223d83d5ed2305dbfd9c60fa54341b67d" Nov 02 21:40:52.368996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:52.368931 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z"] Nov 02 21:40:52.378159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:52.378132 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5935b530-xtw2z"] Nov 02 21:40:52.585732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:52.585694 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db93f7dc-666c-460e-9b4d-672eee6e79c7" path="/var/lib/kubelet/pods/db93f7dc-666c-460e-9b4d-672eee6e79c7/volumes" Nov 02 21:40:52.740281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:52.740150 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:52.889750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:52.888301 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:53.407838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:53.407775 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:53.537895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:53.537844 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:53.777819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:53.777723 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:53.903202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:53.903168 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:54.191997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:54.191847 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:54.208320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:54.208278 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4"] Nov 02 21:40:54.263941 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:54.260629 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e942516f-6qwd4"] Nov 02 21:40:54.537061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:54.534865 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:54.586189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:54.586156 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62382b7-3cef-4b3b-b4b9-4d000b69180f" path="/var/lib/kubelet/pods/d62382b7-3cef-4b3b-b4b9-4d000b69180f/volumes" Nov 02 21:40:55.538097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:55.537924 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:56.285020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:56.284989 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:56.905948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:56.905916 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:57.173080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:57.165884 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:58.692634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:58.692603 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:59.099667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:59.099634 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:59.263330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:59.263301 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:59.397573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:40:59.397495 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:02.231557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:02.231520 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:02.421168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:02.421132 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:02.554609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:02.554520 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:02.733395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:02.733323 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:02.872302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:02.872238 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:03.996508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:03.996477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:05.279725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:05.279687 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj"] Nov 02 21:41:05.280160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:05.279922 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" containerID="cri-o://e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c" gracePeriod=30 Nov 02 21:41:05.326601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:05.326564 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d"] Nov 02 21:41:05.326790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:05.326768 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" containerID="cri-o://cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79" gracePeriod=30 Nov 02 21:41:05.332103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:05.332075 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2"] Nov 02 21:41:05.332317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:05.332286 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" containerID="cri-o://4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09" gracePeriod=30 Nov 02 21:41:06.950049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:06.950013 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:07.183015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:07.182208 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:09.104602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:09.104559 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:09.265446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:09.265406 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:09.450205 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:09.450104 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:10.162235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.162197 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:41:10.162601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.162266 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:41:10.326109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.326081 4441 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:41:10.326214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.326140 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:41:10.661635 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:41:10.661596 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:41:10.674544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.674215 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:41:10.712004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.711940 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:41:10.746769 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.746745 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:41:10.773157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773125 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-kube-api-access-6sfrw\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-kube-api-access-6sfrw\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773181 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-data\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773346 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-podinfo\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773412 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-certs\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773445 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-credential-socket\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773478 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-data" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.773618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773496 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-socket\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773592 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istiod-ca-cert\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773707 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-socket" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.773789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773726 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-credential-socket" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.773789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773743 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-envoy\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.773903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.773811 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-token\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-token\") pod \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\" (UID: \"a79fcb05-a1eb-4999-afa5-673cc8d09e30\") " Nov 02 21:41:10.774116 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.774095 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istiod-ca-cert" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:10.774611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.774591 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.774710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.774614 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.774710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.774630 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.774710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.774643 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.774891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.774838 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-certs" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.776289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.776258 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-kube-api-access-6sfrw" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "kube-api-access-6sfrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.776491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.776321 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-token" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.776491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.776389 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-envoy" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.776822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.776803 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-podinfo" pod "a79fcb05-a1eb-4999-afa5-673cc8d09e30" (UID: "a79fcb05-a1eb-4999-afa5-673cc8d09e30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:10.875704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.875673 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-envoy\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.875725 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-token\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-token\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.875784 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-podinfo\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876269 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-certs\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.876543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876325 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-kube-api-access-rwznl\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-kube-api-access-rwznl\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876469 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-socket\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876516 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-data\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876586 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-kube-api-access-5th5l\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-kube-api-access-5th5l\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.876843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876637 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07c656b-5fbf-4a92-a8f3-214769910deb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07c656b-5fbf-4a92-a8f3-214769910deb-istiod-ca-cert\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.876843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876676 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-data\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.876843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876703 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-socket\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.876843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876738 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-credential-socket\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876787 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-certs\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.876843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876812 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-envoy\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.877268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.876883 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-credential-socket\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.877268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.877068 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-token\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-token\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.877268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.877100 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-podinfo\") pod \"e07c656b-5fbf-4a92-a8f3-214769910deb\" (UID: \"e07c656b-5fbf-4a92-a8f3-214769910deb\") " Nov 02 21:41:10.877268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.877192 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istiod-ca-cert\") pod \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\" (UID: \"81a7c99a-b621-4cdf-80eb-7eef3ab9320c\") " Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.877935 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-socket" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878052 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878076 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878090 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878105 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878117 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfrw\" (UniqueName: \"kubernetes.io/projected/a79fcb05-a1eb-4999-afa5-673cc8d09e30-kube-api-access-6sfrw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878259 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-certs" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.878634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878331 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-credential-socket" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.879136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.878654 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istiod-ca-cert" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:10.879567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.879391 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-data" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.880077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.879821 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07c656b-5fbf-4a92-a8f3-214769910deb-istiod-ca-cert" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:10.880397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.880221 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-data" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.880716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.880533 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-credential-socket" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.881027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.880843 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-socket" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.881446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.881166 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-certs" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.884176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.884129 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-token" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.884491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.884463 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-podinfo" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:10.884774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.884633 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-envoy" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.884774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.884742 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-kube-api-access-rwznl" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "kube-api-access-rwznl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.885087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.885053 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-podinfo" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:10.885087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.885069 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-envoy" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.885206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.885097 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-token" pod "81a7c99a-b621-4cdf-80eb-7eef3ab9320c" (UID: "81a7c99a-b621-4cdf-80eb-7eef3ab9320c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.886782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.886745 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-kube-api-access-5th5l" pod "e07c656b-5fbf-4a92-a8f3-214769910deb" (UID: "e07c656b-5fbf-4a92-a8f3-214769910deb"). InnerVolumeSpecName "kube-api-access-5th5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.980009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.979872 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.979916 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.979932 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.979946 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.979960 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.979997 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980011 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980027 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwznl\" (UniqueName: \"kubernetes.io/projected/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-kube-api-access-rwznl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980042 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980056 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980072 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5th5l\" (UniqueName: \"kubernetes.io/projected/e07c656b-5fbf-4a92-a8f3-214769910deb-kube-api-access-5th5l\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980086 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07c656b-5fbf-4a92-a8f3-214769910deb-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980099 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980113 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980126 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980168 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a7c99a-b621-4cdf-80eb-7eef3ab9320c-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980182 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.980499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:10.980195 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07c656b-5fbf-4a92-a8f3-214769910deb-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.384883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.384851 4441 generic.go:356] "Generic (PLEG): container finished" podID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerID="e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c" exitCode=0 Nov 02 21:41:11.385419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.384909 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" Nov 02 21:41:11.385419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.384932 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" event={"ID":"81a7c99a-b621-4cdf-80eb-7eef3ab9320c","Type":"ContainerDied","Data":"e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c"} Nov 02 21:41:11.385419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.385014 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj" event={"ID":"81a7c99a-b621-4cdf-80eb-7eef3ab9320c","Type":"ContainerDied","Data":"49ad9f93c42cb122425fb451e0442c7e725f1afbdbfe14501159d87d841927b6"} Nov 02 21:41:11.385419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.385050 4441 scope.go:122] "RemoveContainer" containerID="e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c" Nov 02 21:41:11.386611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.386585 4441 generic.go:356] "Generic (PLEG): container finished" podID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerID="4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09" exitCode=0 Nov 02 21:41:11.386833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.386811 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" event={"ID":"e07c656b-5fbf-4a92-a8f3-214769910deb","Type":"ContainerDied","Data":"4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09"} Nov 02 21:41:11.386913 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.386859 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" event={"ID":"e07c656b-5fbf-4a92-a8f3-214769910deb","Type":"ContainerDied","Data":"a2fe4bbd26315748ff6819d461a8d8e0e94b299cfff19a2d8c495abde5fc5541"} Nov 02 21:41:11.387017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.386935 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2" Nov 02 21:41:11.398137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.398121 4441 scope.go:122] "RemoveContainer" containerID="e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c" Nov 02 21:41:11.398421 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:41:11.398395 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c\": container with ID starting with e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c not found: ID does not exist" containerID="e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c" Nov 02 21:41:11.398543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.398431 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c"} err="failed to get container status \"e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c\": rpc error: code = NotFound desc = could not find container \"e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c\": container with ID starting with e84639f4e39141e316e6d7f4bdc9ada8ab89f4be02313a7cc149e06a2c51875c not found: ID does not exist" Nov 02 21:41:11.398543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.398455 4441 scope.go:122] "RemoveContainer" containerID="4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09" Nov 02 21:41:11.398925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.398897 4441 generic.go:356] "Generic (PLEG): container finished" podID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerID="cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79" exitCode=0 Nov 02 21:41:11.399402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.398955 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" Nov 02 21:41:11.399402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.398996 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" event={"ID":"a79fcb05-a1eb-4999-afa5-673cc8d09e30","Type":"ContainerDied","Data":"cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79"} Nov 02 21:41:11.399402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.399044 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d" event={"ID":"a79fcb05-a1eb-4999-afa5-673cc8d09e30","Type":"ContainerDied","Data":"7afc353fe2cec34043fc22f5573bcdd3723601b88440ad07f4fccef026bf689e"} Nov 02 21:41:11.413795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.413776 4441 scope.go:122] "RemoveContainer" containerID="4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09" Nov 02 21:41:11.414260 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:41:11.414233 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09\": container with ID starting with 4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09 not found: ID does not exist" containerID="4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09" Nov 02 21:41:11.414384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.414268 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09"} err="failed to get container status \"4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09\": rpc error: code = NotFound desc = could not find container \"4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09\": container with ID starting with 4f38e7ecc31b4ae3ca3747da14c8e9a1a2c388a0ad002eb97e5137cbdbf28c09 not found: ID does not exist" Nov 02 21:41:11.414384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.414290 4441 scope.go:122] "RemoveContainer" containerID="cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79" Nov 02 21:41:11.427617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.427599 4441 scope.go:122] "RemoveContainer" containerID="cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79" Nov 02 21:41:11.427901 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:41:11.427855 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79\": container with ID starting with cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79 not found: ID does not exist" containerID="cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79" Nov 02 21:41:11.427901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.427883 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79"} err="failed to get container status \"cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79\": rpc error: code = NotFound desc = could not find container \"cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79\": container with ID starting with cbf9b5460d2a87e4a368400caa0d015f17255b9137a9c9666d6ba2d891abbb79 not found: ID does not exist" Nov 02 21:41:11.543567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.543529 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d"] Nov 02 21:41:11.592811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.592730 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-m464d"] Nov 02 21:41:11.697500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.697465 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2"] Nov 02 21:41:11.703126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.703101 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-tvcl2"] Nov 02 21:41:11.819377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.819329 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj"] Nov 02 21:41:11.828031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:11.828005 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-sn5cj"] Nov 02 21:41:12.586797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:12.586756 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" path="/var/lib/kubelet/pods/81a7c99a-b621-4cdf-80eb-7eef3ab9320c/volumes" Nov 02 21:41:12.587519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:12.587493 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" path="/var/lib/kubelet/pods/a79fcb05-a1eb-4999-afa5-673cc8d09e30/volumes" Nov 02 21:41:12.588059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:12.588035 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" path="/var/lib/kubelet/pods/e07c656b-5fbf-4a92-a8f3-214769910deb/volumes" Nov 02 21:41:14.083715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:14.083677 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:19.058138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:19.058101 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:23.787519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:23.787483 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:28.468482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:28.468450 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:28.637239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:28.637210 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:28.833238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:28.833203 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:33.025299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:33.025254 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:33.893206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:33.893167 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:34.262898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:34.262803 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:34.429106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:34.429070 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:38.902707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:38.902670 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:39.059773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:39.056709 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:40.615679 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:41:40.615642 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:41:43.994739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:43.994705 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:45.134269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:45.134233 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:45.534881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:45.534774 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:46.149382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:46.149348 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:46.336458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:46.336422 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:49.195261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:49.195219 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:50.142883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:50.142828 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:50.271266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:50.271230 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:50.406811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:50.406727 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:52.096615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.095943 4441 scope.go:122] "RemoveContainer" containerID="f734db3ea10afe1d592faa65e0c1a79a2c556910f7ca336678e7cef5f143b6ea" Nov 02 21:41:52.117451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.117421 4441 scope.go:122] "RemoveContainer" containerID="31390cf4a1d9738b4a2560d140acb1f5d2dac8fe3fda2d5829ef7187c7b0a94b" Nov 02 21:41:52.127660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.127639 4441 scope.go:122] "RemoveContainer" containerID="dca0c8d8605c6f51a7995d9055973a4e5a9e83b430f153f0d3e132bededd511c" Nov 02 21:41:52.137006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.136982 4441 scope.go:122] "RemoveContainer" containerID="494ce1f380fa07c9695548fc643d6f1b96137b52a3117bd391cb8f2610919aab" Nov 02 21:41:52.145921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.145892 4441 scope.go:122] "RemoveContainer" containerID="3f768160b881840d849021f61f633a7675947fdd204805de7de060c48383026a" Nov 02 21:41:52.159115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.159097 4441 scope.go:122] "RemoveContainer" containerID="1e727870e6a321b307bda108737f84110666c4a777ba22453ca16891215d4878" Nov 02 21:41:52.168183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.168166 4441 scope.go:122] "RemoveContainer" containerID="fcc4394fdff4d6aad65aab380df407151cb1f46b0b409fc0405cde2bd2613fec" Nov 02 21:41:52.179652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.179628 4441 scope.go:122] "RemoveContainer" containerID="74254bbba6181cdd981291816ee79555cef3a76e6171d0ee0300d3944ab77a61" Nov 02 21:41:52.188567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.188548 4441 scope.go:122] "RemoveContainer" containerID="b8d2613684f85ba904713cd8e854aaa913b557cecc6f5d24860ee89381842cd8" Nov 02 21:41:52.198859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.198837 4441 scope.go:122] "RemoveContainer" containerID="75ddfbb745772c3c1498c1221d23b73b339e5e25a01a03ae3659edbd3656ad6d" Nov 02 21:41:52.210099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:52.210082 4441 scope.go:122] "RemoveContainer" containerID="effe2c0698c59582a51b572783dd17dd01370f14cd67d85ee9aadc2ccddffb0d" Nov 02 21:41:53.358159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:53.357046 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:53.566364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:53.566003 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:54.759284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:54.759247 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:57.045956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:57.045917 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k"] Nov 02 21:41:57.114991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:57.114313 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03a9d9-nlm5k"] Nov 02 21:41:58.617992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:58.614445 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a82f95-7b03-415c-81ee-11a3500fb4f5" path="/var/lib/kubelet/pods/a2a82f95-7b03-415c-81ee-11a3500fb4f5/volumes" Nov 02 21:41:59.785994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:59.784547 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:59.931995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:41:59.923580 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:01.598896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:01.598837 4441 patch_prober.go:29] interesting pod/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:42:01.599684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:01.599640 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:42:04.346862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:04.346817 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:04.768563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:04.515003 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:04.768563 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:04.695022 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:06.043425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.043396 4441 generic.go:356] "Generic (PLEG): container finished" podID="367798ae-c8d3-444b-a221-99eec69e544c" containerID="da38b0b3c6255f9ce62b3513a7183f12ae0e157eef368e2b9ae737f0a3dc1e60" exitCode=0 Nov 02 21:42:06.043879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.043477 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" event={"ID":"367798ae-c8d3-444b-a221-99eec69e544c","Type":"ContainerDied","Data":"da38b0b3c6255f9ce62b3513a7183f12ae0e157eef368e2b9ae737f0a3dc1e60"} Nov 02 21:42:06.044031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.044008 4441 scope.go:122] "RemoveContainer" containerID="da38b0b3c6255f9ce62b3513a7183f12ae0e157eef368e2b9ae737f0a3dc1e60" Nov 02 21:42:06.045389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.045364 4441 generic.go:356] "Generic (PLEG): container finished" podID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerID="adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd" exitCode=0 Nov 02 21:42:06.045520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.045433 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" event={"ID":"f1ddcfe4-e1a2-4e98-adee-d30a1297583f","Type":"ContainerDied","Data":"adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd"} Nov 02 21:42:06.045815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.045791 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:42:06.045864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.045811 4441 scope.go:122] "RemoveContainer" containerID="adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd" Nov 02 21:42:06.045864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.045857 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:42:06.045938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.045915 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:42:06.047575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.047547 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:42:06.047644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.047623 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:42:06.047685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:06.047671 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:42:07.150990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.148649 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" event={"ID":"367798ae-c8d3-444b-a221-99eec69e544c","Type":"ContainerStarted","Data":"a9e5517f37ce79e9d17590b250f3e29ffce3d919c6c762116e5bbb639672cf03"} Nov 02 21:42:07.166996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.152979 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" event={"ID":"f1ddcfe4-e1a2-4e98-adee-d30a1297583f","Type":"ContainerStarted","Data":"484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563"} Nov 02 21:42:07.492046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.490635 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:42:07.504991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.497077 4441 patch_prober.go:29] interesting pod/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 02 21:42:07.504991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.497137 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 02 21:42:07.730397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.730142 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:42:07.730397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.730362 4441 patch_prober.go:29] interesting pod/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 21:42:07.730712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:07.730425 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 21:42:08.489559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:08.489466 4441 patch_prober.go:29] interesting pod/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 02 21:42:08.491179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:08.490130 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 02 21:42:08.736003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:08.735151 4441 patch_prober.go:29] interesting pod/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 21:42:08.736003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:08.735212 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 21:42:09.216463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:09.216426 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:09.486329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:09.486238 4441 patch_prober.go:29] interesting pod/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 02 21:42:09.486588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:09.486559 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 02 21:42:09.729578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:09.729540 4441 patch_prober.go:29] interesting pod/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 21:42:09.730167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:09.730124 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 21:42:10.487235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:10.487193 4441 patch_prober.go:29] interesting pod/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 02 21:42:10.487538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:10.487507 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 02 21:42:10.659633 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:42:10.659597 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:42:10.705156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:10.705129 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:42:10.729432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:10.729394 4441 patch_prober.go:29] interesting pod/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 21:42:10.729690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:10.729658 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 21:42:11.490993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:11.485156 4441 patch_prober.go:29] interesting pod/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 02 21:42:11.490993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:11.485215 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 02 21:42:11.739428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:11.738457 4441 patch_prober.go:29] interesting pod/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 21:42:11.739428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:11.738522 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 21:42:12.484800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:12.484760 4441 patch_prober.go:29] interesting pod/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 02 21:42:12.485046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:12.484823 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 02 21:42:12.733793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:12.733761 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:42:12.734212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:12.734192 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:42:12.734986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:12.734929 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:42:13.489099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:13.489070 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:42:13.489928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:13.489908 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:42:13.642131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:13.642098 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:13.808437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:13.808402 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:13.930388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:13.930337 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:18.343996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:18.343941 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:22.800505 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:22.800447 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:25.045726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:25.045691 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x"] Nov 02 21:42:25.092549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:25.092519 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360f33f8-h927x"] Nov 02 21:42:26.585867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:26.585834 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f710b209-cf07-4058-9034-229244973651" path="/var/lib/kubelet/pods/f710b209-cf07-4058-9034-229244973651/volumes" Nov 02 21:42:27.470862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:27.470814 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:29.190004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:29.189937 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:29.621816 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:29.621781 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:29.740439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:29.740405 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:30.357653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:30.357622 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:30.951647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:30.951606 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:32.559933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:32.559899 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:34.827315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:34.827264 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:35.026176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:35.026132 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:35.154657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:35.154573 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:37.482849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:37.482815 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:37.978902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:37.978870 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:38.216757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:38.215732 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:40.615248 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:42:40.615210 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:42:42.347161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:42.347123 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:45.965766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:45.965735 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:47.048339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:47.048308 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:50.486190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:50.486149 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:50.965020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:50.964944 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:42:50.965255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:50.965235 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:42:50.973350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:50.973325 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:42:50.973539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:50.973437 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:42:50.986420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:50.986401 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:42:50.986610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:50.986505 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:42:51.942103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:51.942070 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:52.076435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:52.076396 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:52.335539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:52.335507 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:52.511222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:52.511171 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:57.054004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:57.053960 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:57.231737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:57.231706 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:57.365619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:57.364132 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:57.504047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:57.504013 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:58.186243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:58.186199 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:59.151674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:59.151637 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:59.380938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:42:59.380898 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:00.312133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:00.312100 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:00.695686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:00.695498 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:02.608587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:02.608473 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:02.740140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:02.740108 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:02.875809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:02.875697 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:03.240702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:03.240603 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:03.410231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:03.410192 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:03.539881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:03.539795 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:07.133676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.133644 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:07.320487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.320456 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:07.542652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.542547 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9"] Nov 02 21:43:07.544190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.544158 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" Nov 02 21:43:07.544375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.544360 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" Nov 02 21:43:07.544518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.544506 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" Nov 02 21:43:07.544616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.544604 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" Nov 02 21:43:07.544707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.544697 4441 cpu_manager.go:414] "containerMap: removing container" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" Nov 02 21:43:07.544802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.544790 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" Nov 02 21:43:07.545081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.545063 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07c656b-5fbf-4a92-a8f3-214769910deb" containerName="istio-proxy" Nov 02 21:43:07.545208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.545194 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79fcb05-a1eb-4999-afa5-673cc8d09e30" containerName="istio-proxy" Nov 02 21:43:07.545316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.545302 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a7c99a-b621-4cdf-80eb-7eef3ab9320c" containerName="istio-proxy" Nov 02 21:43:07.548865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.548645 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:07.611010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.603316 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a1bf7d\"" Nov 02 21:43:07.611010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.606807 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a1bf7d-dockercfg-k2vd9\"" Nov 02 21:43:07.611010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.607986 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjm8b\" (UniqueName: \"kubernetes.io/projected/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-kube-api-access-fjm8b\") pod \"payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:07.611010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.608247 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:07.613842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.613802 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:07.617487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.617454 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9"] Nov 02 21:43:07.709078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.709037 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:07.709366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.709350 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjm8b\" (UniqueName: \"kubernetes.io/projected/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-kube-api-access-fjm8b\") pod \"payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:07.710441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.710421 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:07.779643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.779595 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjm8b\" (UniqueName: \"kubernetes.io/projected/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-kube-api-access-fjm8b\") pod \"payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:07.938991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:07.912773 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:08.393476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:08.393447 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9"] Nov 02 21:43:08.398481 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:43:08.398451 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3558ae06_97b2_4aa7_b9a2_93f7183ab76a.slice/crio-7a6a7c25190a69cf71d5feae7c05f74b9dcc04b73b954caa21b4a74b368eb613 WatchSource:0}: Error finding container 7a6a7c25190a69cf71d5feae7c05f74b9dcc04b73b954caa21b4a74b368eb613: Status 404 returned error can't find the container with id 7a6a7c25190a69cf71d5feae7c05f74b9dcc04b73b954caa21b4a74b368eb613 Nov 02 21:43:08.840257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:08.840216 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" event={"ID":"3558ae06-97b2-4aa7-b9a2-93f7183ab76a","Type":"ContainerStarted","Data":"c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933"} Nov 02 21:43:08.840560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:08.840265 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" event={"ID":"3558ae06-97b2-4aa7-b9a2-93f7183ab76a","Type":"ContainerStarted","Data":"7a6a7c25190a69cf71d5feae7c05f74b9dcc04b73b954caa21b4a74b368eb613"} Nov 02 21:43:08.840560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:08.840463 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:10.619764 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:43:10.619720 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:43:12.583734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:12.583699 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:17.340715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:17.340677 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:19.861894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:19.861864 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:43:19.976201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:19.976133 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" podStartSLOduration=12.976113545 podStartE2EDuration="12.976113545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:08.948798781 +0000 UTC m=+1818.504427133" watchObservedRunningTime="2026-11-02 21:43:19.976113545 +0000 UTC m=+1829.531741885" Nov 02 21:43:22.692787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:22.692753 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:27.792553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:27.792510 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:30.188083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:30.188042 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql"] Nov 02 21:43:30.188610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:30.188297 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" containerID="cri-o://3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377" gracePeriod=30 Nov 02 21:43:30.233104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:30.233072 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4"] Nov 02 21:43:30.233338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:30.233314 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" containerID="cri-o://061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21" gracePeriod=30 Nov 02 21:43:32.318033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:32.317895 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:32.648508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:32.648421 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:32.789829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:32.789798 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:33.345735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:33.345696 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:33.526800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:33.526634 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:35.545191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.545164 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:43:35.579802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.579779 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:43:35.701918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.701889 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istiod-ca-cert\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.701918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.701943 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-podinfo\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.702328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.702051 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-token\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-token\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.702456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.702343 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istiod-ca-cert" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:35.702843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.702701 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-credential-socket" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.702962 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.702860 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-credential-socket\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.702962 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.702952 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-podinfo\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.703111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703065 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-envoy\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.703170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703145 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-data\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.703223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703194 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-token\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-token\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703240 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-socket\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703365 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-kube-api-access-hzmqx\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-kube-api-access-hzmqx\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703418 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80264ee9-9160-4b14-a5f8-c4938714fac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80264ee9-9160-4b14-a5f8-c4938714fac4-istiod-ca-cert\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703452 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-kube-api-access-qzqpk\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-kube-api-access-qzqpk\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703481 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-certs\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703516 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-data\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.703579 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-envoy\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.705559 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-certs\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.705587 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80264ee9-9160-4b14-a5f8-c4938714fac4-istiod-ca-cert" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.705612 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-socket\") pod \"80264ee9-9160-4b14-a5f8-c4938714fac4\" (UID: \"80264ee9-9160-4b14-a5f8-c4938714fac4\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.705664 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-credential-socket\") pod \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\" (UID: \"60a378a3-f8dc-48e5-baf8-bb02239ec7ba\") " Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.705684 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-podinfo" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.705996 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-data" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706198 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-certs" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706549 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-credential-socket" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706573 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-envoy" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706761 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-socket" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706639 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706820 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80264ee9-9160-4b14-a5f8-c4938714fac4-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706837 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706852 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706866 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.706881 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.707100 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-data" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.707111 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-certs" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.708724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.707177 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-socket" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.710173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.709133 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-envoy" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:35.710173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.709517 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-token" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:35.712898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.712869 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-kube-api-access-hzmqx" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "kube-api-access-hzmqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:35.713221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.713013 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-podinfo" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:43:35.713497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.713474 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-token" pod "60a378a3-f8dc-48e5-baf8-bb02239ec7ba" (UID: "60a378a3-f8dc-48e5-baf8-bb02239ec7ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:35.714277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.714252 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-kube-api-access-qzqpk" pod "80264ee9-9160-4b14-a5f8-c4938714fac4" (UID: "80264ee9-9160-4b14-a5f8-c4938714fac4"). InnerVolumeSpecName "kube-api-access-qzqpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:35.808747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808720 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808747 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808757 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808766 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808775 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808786 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808801 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808811 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzmqx\" (UniqueName: \"kubernetes.io/projected/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-kube-api-access-hzmqx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808820 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzqpk\" (UniqueName: \"kubernetes.io/projected/80264ee9-9160-4b14-a5f8-c4938714fac4-kube-api-access-qzqpk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808829 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80264ee9-9160-4b14-a5f8-c4938714fac4-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808837 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:35.808915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:35.808845 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a378a3-f8dc-48e5-baf8-bb02239ec7ba-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:36.134552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.134468 4441 generic.go:356] "Generic (PLEG): container finished" podID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerID="061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21" exitCode=0 Nov 02 21:43:36.134552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.134516 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" Nov 02 21:43:36.134746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.134556 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" event={"ID":"60a378a3-f8dc-48e5-baf8-bb02239ec7ba","Type":"ContainerDied","Data":"061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21"} Nov 02 21:43:36.134746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.134609 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4" event={"ID":"60a378a3-f8dc-48e5-baf8-bb02239ec7ba","Type":"ContainerDied","Data":"5102aad8f4ce1c36cbd4266ffac1c11084a6bd7429ae89b00bbaee033ea74c49"} Nov 02 21:43:36.134746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.134641 4441 scope.go:122] "RemoveContainer" containerID="061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21" Nov 02 21:43:36.136087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.136066 4441 generic.go:356] "Generic (PLEG): container finished" podID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerID="3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377" exitCode=0 Nov 02 21:43:36.136220 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.136118 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" Nov 02 21:43:36.136220 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.136150 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" event={"ID":"80264ee9-9160-4b14-a5f8-c4938714fac4","Type":"ContainerDied","Data":"3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377"} Nov 02 21:43:36.136339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.136250 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql" event={"ID":"80264ee9-9160-4b14-a5f8-c4938714fac4","Type":"ContainerDied","Data":"851a5bbd5651d032447086727bb2e26d26ac4a49edac98a911ceef812fb82e38"} Nov 02 21:43:36.146985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.146952 4441 scope.go:122] "RemoveContainer" containerID="061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21" Nov 02 21:43:36.147277 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:43:36.147255 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21\": container with ID starting with 061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21 not found: ID does not exist" containerID="061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21" Nov 02 21:43:36.147374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.147285 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21"} err="failed to get container status \"061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21\": rpc error: code = NotFound desc = could not find container \"061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21\": container with ID starting with 061848241e37324576f9c4f813f039f847b0b361ac9afa1728dc4aa8b6819e21 not found: ID does not exist" Nov 02 21:43:36.147374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.147303 4441 scope.go:122] "RemoveContainer" containerID="3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377" Nov 02 21:43:36.158274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.158259 4441 scope.go:122] "RemoveContainer" containerID="3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377" Nov 02 21:43:36.158534 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:43:36.158516 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377\": container with ID starting with 3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377 not found: ID does not exist" containerID="3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377" Nov 02 21:43:36.158584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.158540 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377"} err="failed to get container status \"3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377\": rpc error: code = NotFound desc = could not find container \"3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377\": container with ID starting with 3c733b88a9d1d3c4a712c66a59a91d775eac6fbbe8e6d24a6fc08c2a94afa377 not found: ID does not exist" Nov 02 21:43:36.293690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.293660 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4"] Nov 02 21:43:36.305804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.305780 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:36.340894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.340867 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kgm4"] Nov 02 21:43:36.402185 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.402103 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql"] Nov 02 21:43:36.406319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.406294 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7g2ql"] Nov 02 21:43:36.425204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.425180 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:36.586119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.586089 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" path="/var/lib/kubelet/pods/60a378a3-f8dc-48e5-baf8-bb02239ec7ba/volumes" Nov 02 21:43:36.586636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:36.586620 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" path="/var/lib/kubelet/pods/80264ee9-9160-4b14-a5f8-c4938714fac4/volumes" Nov 02 21:43:37.954434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:37.954397 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:38.818239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:38.818197 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:39.007133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:39.007099 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:39.137211 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:39.137118 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:39.737629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:39.737596 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l"] Nov 02 21:43:39.737818 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:39.737800 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" podUID="c721ccdd-2315-40fb-9a76-956febbcb546" containerName="discovery" containerID="cri-o://1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067" gracePeriod=30 Nov 02 21:43:39.952104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:39.952069 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:40.069114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.069088 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:43:40.175606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.175575 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-kube-api-access-qmgzp\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-kube-api-access-qmgzp\") pod \"c721ccdd-2315-40fb-9a76-956febbcb546\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " Nov 02 21:43:40.175795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.175640 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-ca-configmap\") pod \"c721ccdd-2315-40fb-9a76-956febbcb546\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " Nov 02 21:43:40.175795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.175736 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c721ccdd-2315-40fb-9a76-956febbcb546-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c721ccdd-2315-40fb-9a76-956febbcb546-local-certs\") pod \"c721ccdd-2315-40fb-9a76-956febbcb546\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " Nov 02 21:43:40.175927 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.175825 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-istio-token\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-istio-token\") pod \"c721ccdd-2315-40fb-9a76-956febbcb546\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " Nov 02 21:43:40.176028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.175942 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-dns-cert\") pod \"c721ccdd-2315-40fb-9a76-956febbcb546\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " Nov 02 21:43:40.176028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.176010 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-cacerts\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-cacerts\") pod \"c721ccdd-2315-40fb-9a76-956febbcb546\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " Nov 02 21:43:40.176169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.176082 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-ca-configmap" pod "c721ccdd-2315-40fb-9a76-956febbcb546" (UID: "c721ccdd-2315-40fb-9a76-956febbcb546"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:40.176169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.176114 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-kubeconfig\") pod \"c721ccdd-2315-40fb-9a76-956febbcb546\" (UID: \"c721ccdd-2315-40fb-9a76-956febbcb546\") " Nov 02 21:43:40.176849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.176821 4441 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-ca-configmap\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:40.178062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.178036 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c721ccdd-2315-40fb-9a76-956febbcb546-local-certs" pod "c721ccdd-2315-40fb-9a76-956febbcb546" (UID: "c721ccdd-2315-40fb-9a76-956febbcb546"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:40.178322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.178301 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-dns-cert" pod "c721ccdd-2315-40fb-9a76-956febbcb546" (UID: "c721ccdd-2315-40fb-9a76-956febbcb546"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:40.178550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.178528 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-istio-token" pod "c721ccdd-2315-40fb-9a76-956febbcb546" (UID: "c721ccdd-2315-40fb-9a76-956febbcb546"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:40.178634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.178575 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-cacerts" pod "c721ccdd-2315-40fb-9a76-956febbcb546" (UID: "c721ccdd-2315-40fb-9a76-956febbcb546"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:40.178634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.178591 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-kubeconfig" pod "c721ccdd-2315-40fb-9a76-956febbcb546" (UID: "c721ccdd-2315-40fb-9a76-956febbcb546"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:40.178634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.178582 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-kube-api-access-qmgzp" pod "c721ccdd-2315-40fb-9a76-956febbcb546" (UID: "c721ccdd-2315-40fb-9a76-956febbcb546"). InnerVolumeSpecName "kube-api-access-qmgzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:40.195262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.195230 4441 generic.go:356] "Generic (PLEG): container finished" podID="c721ccdd-2315-40fb-9a76-956febbcb546" containerID="1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067" exitCode=0 Nov 02 21:43:40.195408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.195308 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" Nov 02 21:43:40.195408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.195325 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" event={"ID":"c721ccdd-2315-40fb-9a76-956febbcb546","Type":"ContainerDied","Data":"1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067"} Nov 02 21:43:40.195408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.195391 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l" event={"ID":"c721ccdd-2315-40fb-9a76-956febbcb546","Type":"ContainerDied","Data":"98a3e8a29f431d2267fe6b9f29a998c5d555805f34a256b1720d837c5373efd6"} Nov 02 21:43:40.195591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.195419 4441 scope.go:122] "RemoveContainer" containerID="1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067" Nov 02 21:43:40.217379 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.217357 4441 scope.go:122] "RemoveContainer" containerID="1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067" Nov 02 21:43:40.217773 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:43:40.217740 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067\": container with ID starting with 1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067 not found: ID does not exist" containerID="1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067" Nov 02 21:43:40.217873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.217783 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067"} err="failed to get container status \"1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067\": rpc error: code = NotFound desc = could not find container \"1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067\": container with ID starting with 1d8f8e01ed5fc0e0f3df1af53ad7d9518fd3f16c5b7bdeb2f892463ec8aaa067 not found: ID does not exist" Nov 02 21:43:40.277902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.277878 4441 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c721ccdd-2315-40fb-9a76-956febbcb546-local-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:40.277902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.277902 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:40.278095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.277913 4441 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-csr-dns-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:40.278095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.277921 4441 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-cacerts\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:40.278095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.277931 4441 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c721ccdd-2315-40fb-9a76-956febbcb546-istio-kubeconfig\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:40.278095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.277939 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmgzp\" (UniqueName: \"kubernetes.io/projected/c721ccdd-2315-40fb-9a76-956febbcb546-kube-api-access-qmgzp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:40.356618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.356589 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l"] Nov 02 21:43:40.360926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.360901 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n2q6l"] Nov 02 21:43:40.401796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.401770 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:40.586481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:40.586439 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c721ccdd-2315-40fb-9a76-956febbcb546" path="/var/lib/kubelet/pods/c721ccdd-2315-40fb-9a76-956febbcb546/volumes" Nov 02 21:43:40.615494 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:43:40.615407 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:43:41.564600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.564555 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9"] Nov 02 21:43:41.565403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565380 4441 cpu_manager.go:414] "containerMap: removing container" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" Nov 02 21:43:41.565532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565404 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" Nov 02 21:43:41.565532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565434 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c721ccdd-2315-40fb-9a76-956febbcb546" containerName="discovery" Nov 02 21:43:41.565532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565441 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c721ccdd-2315-40fb-9a76-956febbcb546" containerName="discovery" Nov 02 21:43:41.565532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565501 4441 cpu_manager.go:414] "containerMap: removing container" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" Nov 02 21:43:41.565532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565510 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" Nov 02 21:43:41.565787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565675 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="80264ee9-9160-4b14-a5f8-c4938714fac4" containerName="istio-proxy" Nov 02 21:43:41.565787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565691 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a378a3-f8dc-48e5-baf8-bb02239ec7ba" containerName="istio-proxy" Nov 02 21:43:41.565787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.565705 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c721ccdd-2315-40fb-9a76-956febbcb546" containerName="discovery" Nov 02 21:43:41.569334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.569291 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.627768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.627737 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-66c72e-openshift-default-dockercfg-gm8g6\"" Nov 02 21:43:41.643310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.643285 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:41.703429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.702931 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-envoy\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703034 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8pj\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-kube-api-access-lf8pj\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703092 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22456806-d042-419a-b631-c348f4775b44-istiod-ca-cert\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703132 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-istio-token\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703244 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-socket\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703415 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-credential-socket\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703443 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-data\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703584 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-certs\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.703983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.703619 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22456806-d042-419a-b631-c348f4775b44-istio-podinfo\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.724314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.724287 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9"] Nov 02 21:43:41.809033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.808998 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.819441 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-credential-socket\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.809853 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-credential-socket\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.822257 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-data\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.822428 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-certs\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.822564 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22456806-d042-419a-b631-c348f4775b44-istio-podinfo\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.822923 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-envoy\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.823076 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8pj\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-kube-api-access-lf8pj\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.823154 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22456806-d042-419a-b631-c348f4775b44-istiod-ca-cert\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.823239 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-istio-token\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.823398 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-socket\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.823898 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-socket\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.824236 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-data\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.833161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.824527 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-certs\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.835719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.835687 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22456806-d042-419a-b631-c348f4775b44-istiod-ca-cert\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.853555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.853426 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22456806-d042-419a-b631-c348f4775b44-istio-podinfo\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.858987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.854331 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-envoy\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.881000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.880928 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8pj\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-kube-api-access-lf8pj\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:41.900650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:41.900617 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-istio-token\") pod \"e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:42.182362 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:42.182263 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:42.617121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:42.617087 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9"] Nov 02 21:43:42.622666 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:43:42.622636 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22456806_d042_419a_b631_c348f4775b44.slice/crio-61b20a750654128f56559f549e91544b2dc96abfb986aa883864bb5138d97584 WatchSource:0}: Error finding container 61b20a750654128f56559f549e91544b2dc96abfb986aa883864bb5138d97584: Status 404 returned error can't find the container with id 61b20a750654128f56559f549e91544b2dc96abfb986aa883864bb5138d97584 Nov 02 21:43:42.625302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:42.625270 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:43:42.625397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:42.625333 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:43:42.625397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:42.625362 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:43:42.828222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:42.828185 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:43.026249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.026153 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:43.151108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.151071 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:43.245993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.229412 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" event={"ID":"22456806-d042-419a-b631-c348f4775b44","Type":"ContainerStarted","Data":"ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d"} Nov 02 21:43:43.245993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.229465 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" event={"ID":"22456806-d042-419a-b631-c348f4775b44","Type":"ContainerStarted","Data":"61b20a750654128f56559f549e91544b2dc96abfb986aa883864bb5138d97584"} Nov 02 21:43:43.245993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.230521 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv"] Nov 02 21:43:43.245993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.234858 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.249989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.247873 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2061072-8d27-49c9-b192-965a7858eaa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.249989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.248111 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jls7w\" (UniqueName: \"kubernetes.io/projected/f2061072-8d27-49c9-b192-965a7858eaa8-kube-api-access-jls7w\") pod \"payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.305044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.301584 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv"] Nov 02 21:43:43.338956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.338238 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-5451a5\"" Nov 02 21:43:43.338956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.338351 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-5451a5-dockercfg-7qvw6\"" Nov 02 21:43:43.374010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.370049 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2061072-8d27-49c9-b192-965a7858eaa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.374010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.370237 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jls7w\" (UniqueName: \"kubernetes.io/projected/f2061072-8d27-49c9-b192-965a7858eaa8-kube-api-access-jls7w\") pod \"payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.374010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.371489 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2061072-8d27-49c9-b192-965a7858eaa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.402989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.376983 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podStartSLOduration=2.376951437 podStartE2EDuration="2.376951437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:43.371800598 +0000 UTC m=+1852.927428936" watchObservedRunningTime="2026-11-02 21:43:43.376951437 +0000 UTC m=+1852.932579774" Nov 02 21:43:43.418056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.414779 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5"] Nov 02 21:43:43.428600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.419626 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:43.470023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.465452 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jls7w\" (UniqueName: \"kubernetes.io/projected/f2061072-8d27-49c9-b192-965a7858eaa8-kube-api-access-jls7w\") pod \"payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.575742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.575572 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:43.581809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.581783 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5"] Nov 02 21:43:43.582698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.582667 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hgz\" (UniqueName: \"kubernetes.io/projected/d11b0d69-980b-46c2-a718-a853071a39c6-kube-api-access-n4hgz\") pod \"payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:43.582854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.582808 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11b0d69-980b-46c2-a718-a853071a39c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:43.686422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.686379 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hgz\" (UniqueName: \"kubernetes.io/projected/d11b0d69-980b-46c2-a718-a853071a39c6-kube-api-access-n4hgz\") pod \"payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:43.686924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.686479 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11b0d69-980b-46c2-a718-a853071a39c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:43.688398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.688372 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11b0d69-980b-46c2-a718-a853071a39c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:43.746371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.745159 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hgz\" (UniqueName: \"kubernetes.io/projected/d11b0d69-980b-46c2-a718-a853071a39c6-kube-api-access-n4hgz\") pod \"payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:43.827146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:43.827070 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:44.040086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.040055 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv"] Nov 02 21:43:44.045390 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:43:44.045359 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2061072_8d27_49c9_b192_965a7858eaa8.slice/crio-e71b721fe13e4637c461dfcfaf0309293fd397f268a20745f1b2fda7345e10a4 WatchSource:0}: Error finding container e71b721fe13e4637c461dfcfaf0309293fd397f268a20745f1b2fda7345e10a4: Status 404 returned error can't find the container with id e71b721fe13e4637c461dfcfaf0309293fd397f268a20745f1b2fda7345e10a4 Nov 02 21:43:44.182919 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.182885 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:44.189109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.189085 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:44.207883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.207859 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5"] Nov 02 21:43:44.211788 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:43:44.211750 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11b0d69_980b_46c2_a718_a853071a39c6.slice/crio-8f21f542efab5ad25e4e8e75c64bfe247ce34a1e8854bc5046a6e71da13cdf70 WatchSource:0}: Error finding container 8f21f542efab5ad25e4e8e75c64bfe247ce34a1e8854bc5046a6e71da13cdf70: Status 404 returned error can't find the container with id 8f21f542efab5ad25e4e8e75c64bfe247ce34a1e8854bc5046a6e71da13cdf70 Nov 02 21:43:44.242432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.242403 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" event={"ID":"d11b0d69-980b-46c2-a718-a853071a39c6","Type":"ContainerStarted","Data":"8f21f542efab5ad25e4e8e75c64bfe247ce34a1e8854bc5046a6e71da13cdf70"} Nov 02 21:43:44.244017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.243991 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" event={"ID":"f2061072-8d27-49c9-b192-965a7858eaa8","Type":"ContainerStarted","Data":"27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698"} Nov 02 21:43:44.244093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.244035 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" event={"ID":"f2061072-8d27-49c9-b192-965a7858eaa8","Type":"ContainerStarted","Data":"e71b721fe13e4637c461dfcfaf0309293fd397f268a20745f1b2fda7345e10a4"} Nov 02 21:43:44.244163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.244148 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:44.244214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.244182 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:44.245076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.245059 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:43:44.529306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:44.529211 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" podStartSLOduration=1.5291949310000001 podStartE2EDuration="1.529194931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:44.526275691 +0000 UTC m=+1854.081904065" watchObservedRunningTime="2026-11-02 21:43:44.529194931 +0000 UTC m=+1854.084823309" Nov 02 21:43:45.254316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:45.254271 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" event={"ID":"d11b0d69-980b-46c2-a718-a853071a39c6","Type":"ContainerStarted","Data":"d7b6691399030695b1e07dcddeda2676e865218c7bf7148a992c54f973e07635"} Nov 02 21:43:45.254887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:45.254863 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:45.394909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:45.394856 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" podStartSLOduration=2.394842557 podStartE2EDuration="2.394842557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:45.389836264 +0000 UTC m=+1854.945464601" watchObservedRunningTime="2026-11-02 21:43:45.394842557 +0000 UTC m=+1854.950470894" Nov 02 21:43:46.142749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:46.142703 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:46.296900 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:46.296869 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:46.424329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:46.424224 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:48.168221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:48.164866 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:49.275795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.275758 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:49.442368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.442303 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:49.702490 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.702454 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp"] Nov 02 21:43:49.707404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.707377 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:49.763437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.763397 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp"] Nov 02 21:43:49.799933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.799153 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c03d4a6-f46b-42bb-9c76-d3386304ed16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:49.799933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.799340 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzddg\" (UniqueName: \"kubernetes.io/projected/5c03d4a6-f46b-42bb-9c76-d3386304ed16-kube-api-access-lzddg\") pod \"payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:49.799933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.799824 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-66c72e-dockercfg-rf7bf\"" Nov 02 21:43:49.799933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.799901 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-66c72e\"" Nov 02 21:43:49.853730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.853702 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8"] Nov 02 21:43:49.857482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.857458 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:49.901885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.901839 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rphk\" (UniqueName: \"kubernetes.io/projected/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-kube-api-access-4rphk\") pod \"payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:49.902273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.902249 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzddg\" (UniqueName: \"kubernetes.io/projected/5c03d4a6-f46b-42bb-9c76-d3386304ed16-kube-api-access-lzddg\") pod \"payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:49.902918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.902886 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:49.903179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.903162 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c03d4a6-f46b-42bb-9c76-d3386304ed16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:49.904272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.904073 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c03d4a6-f46b-42bb-9c76-d3386304ed16-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:49.953416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:49.953327 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8"] Nov 02 21:43:50.013991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.006176 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rphk\" (UniqueName: \"kubernetes.io/projected/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-kube-api-access-4rphk\") pod \"payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:50.013991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.006379 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:50.013991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.007266 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:50.101993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.086776 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzddg\" (UniqueName: \"kubernetes.io/projected/5c03d4a6-f46b-42bb-9c76-d3386304ed16-kube-api-access-lzddg\") pod \"payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:50.101993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.089341 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rphk\" (UniqueName: \"kubernetes.io/projected/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-kube-api-access-4rphk\") pod \"payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:50.177224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.177190 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:50.325112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.325078 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:50.566019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.565993 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8"] Nov 02 21:43:50.571759 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:43:50.571732 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2332601_0ea4_4830_8f4f_dd4a50e5ddc9.slice/crio-01511dd9c1858837c6d47d8c193d5efe8711d8f7ed6e2e5314ae251dc1a8c977 WatchSource:0}: Error finding container 01511dd9c1858837c6d47d8c193d5efe8711d8f7ed6e2e5314ae251dc1a8c977: Status 404 returned error can't find the container with id 01511dd9c1858837c6d47d8c193d5efe8711d8f7ed6e2e5314ae251dc1a8c977 Nov 02 21:43:50.715012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.714978 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp"] Nov 02 21:43:50.724046 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:43:50.724016 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c03d4a6_f46b_42bb_9c76_d3386304ed16.slice/crio-4c2486984229e6f1d4ec05071df90ad94f2b61fe6843485f4d7bf80b2cedacad WatchSource:0}: Error finding container 4c2486984229e6f1d4ec05071df90ad94f2b61fe6843485f4d7bf80b2cedacad: Status 404 returned error can't find the container with id 4c2486984229e6f1d4ec05071df90ad94f2b61fe6843485f4d7bf80b2cedacad Nov 02 21:43:50.830438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:50.830364 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:51.321364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.321328 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" event={"ID":"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9","Type":"ContainerStarted","Data":"6e8938ae4e9f076f2c7a154a39185009611ef7273e9975e40cacf182f6c29fb4"} Nov 02 21:43:51.321557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.321396 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:43:51.321557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.321417 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" event={"ID":"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9","Type":"ContainerStarted","Data":"01511dd9c1858837c6d47d8c193d5efe8711d8f7ed6e2e5314ae251dc1a8c977"} Nov 02 21:43:51.323139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.323111 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" event={"ID":"5c03d4a6-f46b-42bb-9c76-d3386304ed16","Type":"ContainerStarted","Data":"6e7399cc362cd05811872222afe7230379852e28ca775201ced738944c716c54"} Nov 02 21:43:51.323282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.323150 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" event={"ID":"5c03d4a6-f46b-42bb-9c76-d3386304ed16","Type":"ContainerStarted","Data":"4c2486984229e6f1d4ec05071df90ad94f2b61fe6843485f4d7bf80b2cedacad"} Nov 02 21:43:51.323282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.323170 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:43:51.434347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.434291 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" podStartSLOduration=2.434273482 podStartE2EDuration="2.434273482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:51.427829215 +0000 UTC m=+1860.983457552" watchObservedRunningTime="2026-11-02 21:43:51.434273482 +0000 UTC m=+1860.989901820" Nov 02 21:43:51.535263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:51.535200 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" podStartSLOduration=2.535187954 podStartE2EDuration="2.535187954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:51.534035387 +0000 UTC m=+1861.089663726" watchObservedRunningTime="2026-11-02 21:43:51.535187954 +0000 UTC m=+1861.090816291" Nov 02 21:43:52.337386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:52.337357 4441 scope.go:122] "RemoveContainer" containerID="c314de03ceda908fa3eaf08bee79b4d2030d9b239ac5bbd5d2925a015b653501" Nov 02 21:43:52.348619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:52.348598 4441 scope.go:122] "RemoveContainer" containerID="df8938677ad54d2af71dd63bc37d552f12685fd84eeb6d5085e4fb06009df330" Nov 02 21:43:52.358135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:52.358115 4441 scope.go:122] "RemoveContainer" containerID="920d19ca5f9bd2767307b5ab5fcffa009efd79afbc7723d568869023b555461f" Nov 02 21:43:52.370749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:52.370730 4441 scope.go:122] "RemoveContainer" containerID="cf0e0a6fc2e1b8ddb4999e722e9f311ffc9eb84e7a361ccbc4a8beed27516a1c" Nov 02 21:43:52.380682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:52.380661 4441 scope.go:122] "RemoveContainer" containerID="8ad5aaa82c398e5631f8a2265d9f0473c209aeaf534321c1681c667920ebfdce" Nov 02 21:43:53.104414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:53.104379 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:55.257946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:55.257916 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:43:56.270458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:56.270426 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:43:57.872105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:57.872074 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:58.012630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:58.012599 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:58.132049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:43:58.131451 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:02.333906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:02.333874 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:44:02.334416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:02.333961 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:44:02.978713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:02.978681 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:05.108778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:05.108742 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:05.226146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:05.226110 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:08.000465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:08.000435 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:10.615667 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:44:10.615627 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:44:12.917155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:12.917117 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:17.637181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:17.637142 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:19.310593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.310560 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7167-3e3a3b"] Nov 02 21:44:19.315196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.315175 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:19.365449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.365419 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7167-3e3a3b"] Nov 02 21:44:19.405441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.405412 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9vlh9\"" Nov 02 21:44:19.408442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.408406 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wcv9\" (UniqueName: \"kubernetes.io/projected/73f93626-f2cb-44dc-8a67-8097eb2c1c72-kube-api-access-5wcv9\") pod \"test-curl-7167-3e3a3b\" (UID: \"73f93626-f2cb-44dc-8a67-8097eb2c1c72\") " pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:19.509558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.509528 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wcv9\" (UniqueName: \"kubernetes.io/projected/73f93626-f2cb-44dc-8a67-8097eb2c1c72-kube-api-access-5wcv9\") pod \"test-curl-7167-3e3a3b\" (UID: \"73f93626-f2cb-44dc-8a67-8097eb2c1c72\") " pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:19.563540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.563473 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wcv9\" (UniqueName: \"kubernetes.io/projected/73f93626-f2cb-44dc-8a67-8097eb2c1c72-kube-api-access-5wcv9\") pod \"test-curl-7167-3e3a3b\" (UID: \"73f93626-f2cb-44dc-8a67-8097eb2c1c72\") " pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:19.629133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:19.629110 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:20.001933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:20.001855 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7167-3e3a3b"] Nov 02 21:44:20.006719 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:44:20.006687 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f93626_f2cb_44dc_8a67_8097eb2c1c72.slice/crio-c43730cf5c634a9dedd65638b921e7e1a79cb10e53b3d1ebece58729731b4e06 WatchSource:0}: Error finding container c43730cf5c634a9dedd65638b921e7e1a79cb10e53b3d1ebece58729731b4e06: Status 404 returned error can't find the container with id c43730cf5c634a9dedd65638b921e7e1a79cb10e53b3d1ebece58729731b4e06 Nov 02 21:44:20.008717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:20.008700 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:44:20.584484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:20.584440 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7167-3e3a3b" event={"ID":"73f93626-f2cb-44dc-8a67-8097eb2c1c72","Type":"ContainerStarted","Data":"c43730cf5c634a9dedd65638b921e7e1a79cb10e53b3d1ebece58729731b4e06"} Nov 02 21:44:22.252404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:22.252369 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:22.401716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:22.401654 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:22.533508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:22.533409 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:23.606560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:23.606523 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7167-3e3a3b" event={"ID":"73f93626-f2cb-44dc-8a67-8097eb2c1c72","Type":"ContainerStarted","Data":"c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865"} Nov 02 21:44:23.714440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:23.714393 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7167-3e3a3b" podStartSLOduration=1.977267238 podStartE2EDuration="4.714380564s" totalImagesPullingTime="2.737113326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:23.713006329 +0000 UTC m=+1893.268634667" watchObservedRunningTime="2026-11-02 21:44:23.714380564 +0000 UTC m=+1893.270008901" Nov 02 21:44:25.291059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:25.291022 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7167-3e3a3b"] Nov 02 21:44:25.298994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:25.298934 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7167-3e3a3b"] Nov 02 21:44:25.622848 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:25.622808 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7167-3e3a3b" podUID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" containerName="test-curl-7167-3e3a3b" containerID="cri-o://c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865" gracePeriod=2 Nov 02 21:44:25.672154 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:44:25.672109 4441 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7167-3e3a3b\" is forbidden: User \"system:node:ip-10-0-137-225.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-225.us-west-2.compute.internal' and this object" podUID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:26.125561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.125528 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk"] Nov 02 21:44:26.126611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.126585 4441 cpu_manager.go:414] "containerMap: removing container" podUID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" containerName="test-curl-7167-3e3a3b" Nov 02 21:44:26.126611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.126610 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" containerName="test-curl-7167-3e3a3b" Nov 02 21:44:26.126826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.126783 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" containerName="test-curl-7167-3e3a3b" Nov 02 21:44:26.132013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.131947 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.180844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.180813 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk"] Nov 02 21:44:26.181113 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:44:26.181003 4441 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7167-3e3a3b\" is forbidden: User \"system:node:ip-10-0-137-225.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-225.us-west-2.compute.internal' and this object" podUID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:26.213184 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.213148 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.213375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.213349 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.213550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.213522 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rp7q\" (UniqueName: \"kubernetes.io/projected/c7b39b84-c2fa-4701-bb98-224e7fab65aa-kube-api-access-8rp7q\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.221155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.221122 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:44:26.221155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.221141 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:44:26.221343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.221153 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:44:26.221343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.221266 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:44:26.315150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.315115 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.315644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.315208 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rp7q\" (UniqueName: \"kubernetes.io/projected/c7b39b84-c2fa-4701-bb98-224e7fab65aa-kube-api-access-8rp7q\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.315644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.315361 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.316052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.316030 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.317523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.317499 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.371498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.371473 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rp7q\" (UniqueName: \"kubernetes.io/projected/c7b39b84-c2fa-4701-bb98-224e7fab65aa-kube-api-access-8rp7q\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.449259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.449193 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:26.829170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:26.829132 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk"] Nov 02 21:44:26.835266 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:44:26.835241 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b39b84_c2fa_4701_bb98_224e7fab65aa.slice/crio-50dedfff3b8a140adab61e8c76ffb67cdaf59b001145d3e38b78233da1e727e7 WatchSource:0}: Error finding container 50dedfff3b8a140adab61e8c76ffb67cdaf59b001145d3e38b78233da1e727e7: Status 404 returned error can't find the container with id 50dedfff3b8a140adab61e8c76ffb67cdaf59b001145d3e38b78233da1e727e7 Nov 02 21:44:27.091303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:27.091255 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:27.641128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:27.641094 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" event={"ID":"c7b39b84-c2fa-4701-bb98-224e7fab65aa","Type":"ContainerStarted","Data":"98a7a5710778d3496e1ec261419a32e97e230aaf3e4843fee308c98ca0ed134f"} Nov 02 21:44:27.641456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:27.641154 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" event={"ID":"c7b39b84-c2fa-4701-bb98-224e7fab65aa","Type":"ContainerStarted","Data":"50dedfff3b8a140adab61e8c76ffb67cdaf59b001145d3e38b78233da1e727e7"} Nov 02 21:44:27.751961 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:27.751895 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" podStartSLOduration=1.751876997 podStartE2EDuration="1.751876997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:27.74759871 +0000 UTC m=+1897.303227047" watchObservedRunningTime="2026-11-02 21:44:27.751876997 +0000 UTC m=+1897.307505338" Nov 02 21:44:27.955704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:27.955678 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:28.039195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.039163 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f93626-f2cb-44dc-8a67-8097eb2c1c72-kube-api-access-5wcv9\" (UniqueName: \"kubernetes.io/projected/73f93626-f2cb-44dc-8a67-8097eb2c1c72-kube-api-access-5wcv9\") pod \"73f93626-f2cb-44dc-8a67-8097eb2c1c72\" (UID: \"73f93626-f2cb-44dc-8a67-8097eb2c1c72\") " Nov 02 21:44:28.041295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.041274 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f93626-f2cb-44dc-8a67-8097eb2c1c72-kube-api-access-5wcv9" pod "73f93626-f2cb-44dc-8a67-8097eb2c1c72" (UID: "73f93626-f2cb-44dc-8a67-8097eb2c1c72"). InnerVolumeSpecName "kube-api-access-5wcv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:28.141670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.141597 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wcv9\" (UniqueName: \"kubernetes.io/projected/73f93626-f2cb-44dc-8a67-8097eb2c1c72-kube-api-access-5wcv9\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:28.589330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.589297 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" path="/var/lib/kubelet/pods/73f93626-f2cb-44dc-8a67-8097eb2c1c72/volumes" Nov 02 21:44:28.655218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.655186 4441 generic.go:356] "Generic (PLEG): container finished" podID="c7b39b84-c2fa-4701-bb98-224e7fab65aa" containerID="98a7a5710778d3496e1ec261419a32e97e230aaf3e4843fee308c98ca0ed134f" exitCode=0 Nov 02 21:44:28.655636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.655276 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" event={"ID":"c7b39b84-c2fa-4701-bb98-224e7fab65aa","Type":"ContainerDied","Data":"98a7a5710778d3496e1ec261419a32e97e230aaf3e4843fee308c98ca0ed134f"} Nov 02 21:44:28.656846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.656824 4441 generic.go:356] "Generic (PLEG): container finished" podID="73f93626-f2cb-44dc-8a67-8097eb2c1c72" containerID="c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865" exitCode=137 Nov 02 21:44:28.656953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.656855 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-3e3a3b" Nov 02 21:44:28.656953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.656894 4441 scope.go:122] "RemoveContainer" containerID="c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865" Nov 02 21:44:28.667136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.667119 4441 scope.go:122] "RemoveContainer" containerID="c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865" Nov 02 21:44:28.667385 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:44:28.667366 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865\": container with ID starting with c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865 not found: ID does not exist" containerID="c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865" Nov 02 21:44:28.667450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:28.667393 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865"} err="failed to get container status \"c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865\": rpc error: code = NotFound desc = could not find container \"c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865\": container with ID starting with c22e4e5cec9e35b7cb497527ed9d29a3afdc7346a7aeb2e4913eda2ce2770865 not found: ID does not exist" Nov 02 21:44:29.837636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.837611 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:29.970910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.970831 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-xks-ca\") pod \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " Nov 02 21:44:29.970910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.970895 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-openshift-ca\") pod \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " Nov 02 21:44:29.971175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.971069 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b39b84-c2fa-4701-bb98-224e7fab65aa-kube-api-access-8rp7q\" (UniqueName: \"kubernetes.io/projected/c7b39b84-c2fa-4701-bb98-224e7fab65aa-kube-api-access-8rp7q\") pod \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\" (UID: \"c7b39b84-c2fa-4701-bb98-224e7fab65aa\") " Nov 02 21:44:29.971353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.971331 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-openshift-ca" pod "c7b39b84-c2fa-4701-bb98-224e7fab65aa" (UID: "c7b39b84-c2fa-4701-bb98-224e7fab65aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:44:29.971749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.971723 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:29.973396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.973371 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b39b84-c2fa-4701-bb98-224e7fab65aa-kube-api-access-8rp7q" pod "c7b39b84-c2fa-4701-bb98-224e7fab65aa" (UID: "c7b39b84-c2fa-4701-bb98-224e7fab65aa"). InnerVolumeSpecName "kube-api-access-8rp7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:29.973750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:29.973729 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-xks-ca" pod "c7b39b84-c2fa-4701-bb98-224e7fab65aa" (UID: "c7b39b84-c2fa-4701-bb98-224e7fab65aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:30.073347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:30.073317 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b39b84-c2fa-4701-bb98-224e7fab65aa-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:30.073347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:30.073345 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rp7q\" (UniqueName: \"kubernetes.io/projected/c7b39b84-c2fa-4701-bb98-224e7fab65aa-kube-api-access-8rp7q\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:30.678886 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:30.678850 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" event={"ID":"c7b39b84-c2fa-4701-bb98-224e7fab65aa","Type":"ContainerDied","Data":"50dedfff3b8a140adab61e8c76ffb67cdaf59b001145d3e38b78233da1e727e7"} Nov 02 21:44:30.679067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:30.678901 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50dedfff3b8a140adab61e8c76ffb67cdaf59b001145d3e38b78233da1e727e7" Nov 02 21:44:30.679067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:30.678859 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk" Nov 02 21:44:31.887134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:31.887104 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:32.066530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:32.066493 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:32.187125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:32.187009 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:36.933174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:36.933139 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:38.887232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:38.887188 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:39.007793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:39.007756 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:40.617018 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:44:40.616959 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:44:42.171609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:42.171565 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:43.859906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:43.859875 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:45.131486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:45.131449 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:45.264115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:45.264083 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:45.419751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:45.419663 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:46.580173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:46.580135 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:46.711932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:46.711899 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:46.848484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:46.848384 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:47.490610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:47.490569 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:51.175417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:51.175376 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:51.292702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:51.292675 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:52.441156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:52.441118 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:53.757322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.757288 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5"] Nov 02 21:44:53.758019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.758003 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c7b39b84-c2fa-4701-bb98-224e7fab65aa" containerName="revoke-keys" Nov 02 21:44:53.758019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.758019 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b39b84-c2fa-4701-bb98-224e7fab65aa" containerName="revoke-keys" Nov 02 21:44:53.758179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.758167 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b39b84-c2fa-4701-bb98-224e7fab65aa" containerName="revoke-keys" Nov 02 21:44:53.762897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.762880 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.816899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.816868 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5"] Nov 02 21:44:53.856815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.856777 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:44:53.856995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.856827 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:44:53.859402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.859378 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:44:53.859507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.859419 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:44:53.869337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.869314 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/cbdd20b1-14eb-485c-8a7f-d1fd42660334-kube-api-access-fnwxn\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.869548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.869529 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.869602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.869578 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.970814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.970782 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/cbdd20b1-14eb-485c-8a7f-d1fd42660334-kube-api-access-fnwxn\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.971004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.970904 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.971004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.970932 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.971736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.971711 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:53.973410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:53.973380 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:54.025053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:54.024984 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/cbdd20b1-14eb-485c-8a7f-d1fd42660334-kube-api-access-fnwxn\") pod \"maas-api-revoke-sub-a0689b4f-slgl5\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:54.077275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:54.077243 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:54.454352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:54.454321 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5"] Nov 02 21:44:54.460194 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:44:54.460166 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdd20b1_14eb_485c_8a7f_d1fd42660334.slice/crio-442e31381282d9953799d4ab2f6323e74663860907b2864000ca1c8bf5f8d341 WatchSource:0}: Error finding container 442e31381282d9953799d4ab2f6323e74663860907b2864000ca1c8bf5f8d341: Status 404 returned error can't find the container with id 442e31381282d9953799d4ab2f6323e74663860907b2864000ca1c8bf5f8d341 Nov 02 21:44:54.910688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:54.910651 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" event={"ID":"cbdd20b1-14eb-485c-8a7f-d1fd42660334","Type":"ContainerStarted","Data":"95c7e699b2c1c60b22b7d48e31c71572539a5d39910763a7321fe5ec793af3fd"} Nov 02 21:44:54.911276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:54.911240 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" event={"ID":"cbdd20b1-14eb-485c-8a7f-d1fd42660334","Type":"ContainerStarted","Data":"442e31381282d9953799d4ab2f6323e74663860907b2864000ca1c8bf5f8d341"} Nov 02 21:44:55.019583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:55.019530 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" podStartSLOduration=2.01951803 podStartE2EDuration="2.01951803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:55.01759601 +0000 UTC m=+1924.573224341" watchObservedRunningTime="2026-11-02 21:44:55.01951803 +0000 UTC m=+1924.575146366" Nov 02 21:44:55.920820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:55.920789 4441 generic.go:356] "Generic (PLEG): container finished" podID="cbdd20b1-14eb-485c-8a7f-d1fd42660334" containerID="95c7e699b2c1c60b22b7d48e31c71572539a5d39910763a7321fe5ec793af3fd" exitCode=0 Nov 02 21:44:55.921247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:55.920870 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" event={"ID":"cbdd20b1-14eb-485c-8a7f-d1fd42660334","Type":"ContainerDied","Data":"95c7e699b2c1c60b22b7d48e31c71572539a5d39910763a7321fe5ec793af3fd"} Nov 02 21:44:57.130207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.130181 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:57.214027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.213925 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdd20b1-14eb-485c-8a7f-d1fd42660334-kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/cbdd20b1-14eb-485c-8a7f-d1fd42660334-kube-api-access-fnwxn\") pod \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " Nov 02 21:44:57.214195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.214142 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-openshift-ca\") pod \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " Nov 02 21:44:57.214271 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.214246 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-xks-ca\") pod \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\" (UID: \"cbdd20b1-14eb-485c-8a7f-d1fd42660334\") " Nov 02 21:44:57.214579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.214553 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-openshift-ca" pod "cbdd20b1-14eb-485c-8a7f-d1fd42660334" (UID: "cbdd20b1-14eb-485c-8a7f-d1fd42660334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:44:57.214917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.214892 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:57.216454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.216429 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-xks-ca" pod "cbdd20b1-14eb-485c-8a7f-d1fd42660334" (UID: "cbdd20b1-14eb-485c-8a7f-d1fd42660334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:57.216872 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.216848 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdd20b1-14eb-485c-8a7f-d1fd42660334-kube-api-access-fnwxn" pod "cbdd20b1-14eb-485c-8a7f-d1fd42660334" (UID: "cbdd20b1-14eb-485c-8a7f-d1fd42660334"). InnerVolumeSpecName "kube-api-access-fnwxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:57.271010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.270938 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:57.316777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.316736 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbdd20b1-14eb-485c-8a7f-d1fd42660334-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:57.316777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.316774 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/cbdd20b1-14eb-485c-8a7f-d1fd42660334-kube-api-access-fnwxn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:57.939047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.939011 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" event={"ID":"cbdd20b1-14eb-485c-8a7f-d1fd42660334","Type":"ContainerDied","Data":"442e31381282d9953799d4ab2f6323e74663860907b2864000ca1c8bf5f8d341"} Nov 02 21:44:57.939047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.939045 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5" Nov 02 21:44:57.939269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:44:57.939070 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442e31381282d9953799d4ab2f6323e74663860907b2864000ca1c8bf5f8d341" Nov 02 21:45:00.161247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.161211 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8"] Nov 02 21:45:00.162110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.162086 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cbdd20b1-14eb-485c-8a7f-d1fd42660334" containerName="revoke-keys" Nov 02 21:45:00.162110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.162109 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdd20b1-14eb-485c-8a7f-d1fd42660334" containerName="revoke-keys" Nov 02 21:45:00.162326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.162312 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdd20b1-14eb-485c-8a7f-d1fd42660334" containerName="revoke-keys" Nov 02 21:45:00.166537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.166514 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4"] Nov 02 21:45:00.166652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.166630 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.171261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.171237 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt"] Nov 02 21:45:00.171375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.171333 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.176003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.175983 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.211282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.211254 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7"] Nov 02 21:45:00.216020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.215956 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.218659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.218635 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj"] Nov 02 21:45:00.224173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.224124 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.224775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.224755 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms"] Nov 02 21:45:00.229202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.229178 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.229337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.229242 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2"] Nov 02 21:45:00.234493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.234370 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.234842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.234819 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt"] Nov 02 21:45:00.240516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.240494 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.241025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.240962 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz"] Nov 02 21:45:00.245476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.245457 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.245807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.245781 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh"] Nov 02 21:45:00.251228 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.251205 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:45:00.251345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.251206 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:00.252851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.252830 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:00.257225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257206 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d"] Nov 02 21:45:00.257352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257334 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.257422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257347 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cm69\" (UniqueName: \"kubernetes.io/projected/6e5862b7-d0b9-44df-ab25-5c562cae664a-kube-api-access-2cm69\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.257422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257386 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc41514-68ad-4d9d-bb6c-9ccf26358703-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.257519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257496 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c376f194-53a4-4d9f-bf63-dadcdbf79542-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.257609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257588 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.257647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257637 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2c6d\" (UniqueName: \"kubernetes.io/projected/d3e37857-599f-4d61-9908-0d0b57269498-kube-api-access-n2c6d\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.257694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.257662 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757f3a79-c1cb-46c8-8210-bee23a846f45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.258169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258150 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv88l\" (UniqueName: \"kubernetes.io/projected/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-kube-api-access-vv88l\") pod \"maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.258311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258197 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e37857-599f-4d61-9908-0d0b57269498-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.258311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258246 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c34b-ff3f-40d6-afad-3172880687f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.258441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258327 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mp5n\" (UniqueName: \"kubernetes.io/projected/53142d59-d966-4f70-9609-f8613c9b2547-kube-api-access-7mp5n\") pod \"maas-api-key-cleanup-29894265-ckwms\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.258441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258379 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f6q4\" (UniqueName: \"kubernetes.io/projected/0bc41514-68ad-4d9d-bb6c-9ccf26358703-kube-api-access-7f6q4\") pod \"maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.258441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258426 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53142d59-d966-4f70-9609-f8613c9b2547-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-ckwms\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.258617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258500 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrmn\" (UniqueName: \"kubernetes.io/projected/8aa2c34b-ff3f-40d6-afad-3172880687f7-kube-api-access-znrmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.258617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258548 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhz68\" (UniqueName: \"kubernetes.io/projected/c376f194-53a4-4d9f-bf63-dadcdbf79542-kube-api-access-qhz68\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.258925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.258901 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5862b7-d0b9-44df-ab25-5c562cae664a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.259026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.259007 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjprj\" (UniqueName: \"kubernetes.io/projected/757f3a79-c1cb-46c8-8210-bee23a846f45-kube-api-access-bjprj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.262010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.261958 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.262488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.262477 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7"] Nov 02 21:45:00.266994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.266948 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.267092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.266999 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx"] Nov 02 21:45:00.271715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.271695 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.272047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.272027 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8"] Nov 02 21:45:00.278404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.278357 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4"] Nov 02 21:45:00.322083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.322046 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj"] Nov 02 21:45:00.340636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.340607 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7"] Nov 02 21:45:00.391105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.390947 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhz68\" (UniqueName: \"kubernetes.io/projected/c376f194-53a4-4d9f-bf63-dadcdbf79542-kube-api-access-qhz68\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.391394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.391368 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5862b7-d0b9-44df-ab25-5c562cae664a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.391695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.391673 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjprj\" (UniqueName: \"kubernetes.io/projected/757f3a79-c1cb-46c8-8210-bee23a846f45-kube-api-access-bjprj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.391848 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.391829 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cm69\" (UniqueName: \"kubernetes.io/projected/6e5862b7-d0b9-44df-ab25-5c562cae664a-kube-api-access-2cm69\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.391955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.391933 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc41514-68ad-4d9d-bb6c-9ccf26358703-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.392099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392013 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c376f194-53a4-4d9f-bf63-dadcdbf79542-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.392099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392050 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dae9676-856a-44a5-904e-e32d2a114e28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.392214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392129 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.392214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392187 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2c6d\" (UniqueName: \"kubernetes.io/projected/d3e37857-599f-4d61-9908-0d0b57269498-kube-api-access-n2c6d\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.392322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392216 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757f3a79-c1cb-46c8-8210-bee23a846f45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.392322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392245 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv88l\" (UniqueName: \"kubernetes.io/projected/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-kube-api-access-vv88l\") pod \"maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.392322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392278 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e37857-599f-4d61-9908-0d0b57269498-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.392486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392343 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c34b-ff3f-40d6-afad-3172880687f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.392486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392372 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlnb\" (UniqueName: \"kubernetes.io/projected/096f83eb-1617-4613-b023-864f9cb54f53-kube-api-access-mzlnb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.392486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392439 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mp5n\" (UniqueName: \"kubernetes.io/projected/53142d59-d966-4f70-9609-f8613c9b2547-kube-api-access-7mp5n\") pod \"maas-api-key-cleanup-29894265-ckwms\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.392486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392473 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6q4\" (UniqueName: \"kubernetes.io/projected/0bc41514-68ad-4d9d-bb6c-9ccf26358703-kube-api-access-7f6q4\") pod \"maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.392706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392503 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7sw\" (UniqueName: \"kubernetes.io/projected/6dae9676-856a-44a5-904e-e32d2a114e28-kube-api-access-xl7sw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.392706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392534 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53142d59-d966-4f70-9609-f8613c9b2547-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-ckwms\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.392706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392609 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096f83eb-1617-4613-b023-864f9cb54f53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.392706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392674 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc85c55-68e7-40d8-b830-eee11c38de6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.392914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392716 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsk8l\" (UniqueName: \"kubernetes.io/projected/bfc85c55-68e7-40d8-b830-eee11c38de6c-kube-api-access-vsk8l\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.392914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.392747 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrmn\" (UniqueName: \"kubernetes.io/projected/8aa2c34b-ff3f-40d6-afad-3172880687f7-kube-api-access-znrmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.396556 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.396529 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5862b7-d0b9-44df-ab25-5c562cae664a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.398257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.398231 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e37857-599f-4d61-9908-0d0b57269498-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.398885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.398862 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c34b-ff3f-40d6-afad-3172880687f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.405504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.402040 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.405504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.402835 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c376f194-53a4-4d9f-bf63-dadcdbf79542-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.405504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.403819 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt"] Nov 02 21:45:00.405504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.403857 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53142d59-d966-4f70-9609-f8613c9b2547-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-ckwms\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.405504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.405498 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757f3a79-c1cb-46c8-8210-bee23a846f45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.405829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.405540 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc41514-68ad-4d9d-bb6c-9ccf26358703-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.427137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.427116 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh"] Nov 02 21:45:00.432195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.432073 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz"] Nov 02 21:45:00.437568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.437542 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d"] Nov 02 21:45:00.443747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.443706 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7"] Nov 02 21:45:00.447331 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.447310 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrmn\" (UniqueName: \"kubernetes.io/projected/8aa2c34b-ff3f-40d6-afad-3172880687f7-kube-api-access-znrmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.447891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.447875 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms"] Nov 02 21:45:00.449700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.449656 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv88l\" (UniqueName: \"kubernetes.io/projected/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-kube-api-access-vv88l\") pod \"maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.453224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.453197 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhz68\" (UniqueName: \"kubernetes.io/projected/c376f194-53a4-4d9f-bf63-dadcdbf79542-kube-api-access-qhz68\") pod \"maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.454209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.454152 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt"] Nov 02 21:45:00.459374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.459352 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6q4\" (UniqueName: \"kubernetes.io/projected/0bc41514-68ad-4d9d-bb6c-9ccf26358703-kube-api-access-7f6q4\") pod \"maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.461614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.461084 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cm69\" (UniqueName: \"kubernetes.io/projected/6e5862b7-d0b9-44df-ab25-5c562cae664a-kube-api-access-2cm69\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.461614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.461546 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2"] Nov 02 21:45:00.463349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.463325 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mp5n\" (UniqueName: \"kubernetes.io/projected/53142d59-d966-4f70-9609-f8613c9b2547-kube-api-access-7mp5n\") pod \"maas-api-key-cleanup-29894265-ckwms\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.463874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.463852 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjprj\" (UniqueName: \"kubernetes.io/projected/757f3a79-c1cb-46c8-8210-bee23a846f45-kube-api-access-bjprj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.473417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.473394 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx"] Nov 02 21:45:00.495380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.495182 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:00.497006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.496947 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2c6d\" (UniqueName: \"kubernetes.io/projected/d3e37857-599f-4d61-9908-0d0b57269498-kube-api-access-n2c6d\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.503572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.503543 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:00.508170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508145 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgprl\" (UniqueName: \"kubernetes.io/projected/0a1aae5a-96d6-43b1-83cb-85d3a097faea-kube-api-access-dgprl\") pod \"maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.508276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508194 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dae9676-856a-44a5-904e-e32d2a114e28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.508276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508215 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48312fad-cf56-4728-8f43-ad84bad2537d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.508433 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508411 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlnb\" (UniqueName: \"kubernetes.io/projected/096f83eb-1617-4613-b023-864f9cb54f53-kube-api-access-mzlnb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.508505 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508489 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7sw\" (UniqueName: \"kubernetes.io/projected/6dae9676-856a-44a5-904e-e32d2a114e28-kube-api-access-xl7sw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.508566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508550 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096f83eb-1617-4613-b023-864f9cb54f53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.508630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508603 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc85c55-68e7-40d8-b830-eee11c38de6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.508694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508637 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsk8l\" (UniqueName: \"kubernetes.io/projected/bfc85c55-68e7-40d8-b830-eee11c38de6c-kube-api-access-vsk8l\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.508694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508684 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1aae5a-96d6-43b1-83cb-85d3a097faea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.508792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508775 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5vt\" (UniqueName: \"kubernetes.io/projected/48312fad-cf56-4728-8f43-ad84bad2537d-kube-api-access-bv5vt\") pod \"maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.508937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.508918 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dae9676-856a-44a5-904e-e32d2a114e28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.510628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.510078 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096f83eb-1617-4613-b023-864f9cb54f53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.510628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.510082 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc85c55-68e7-40d8-b830-eee11c38de6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.533363 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.533336 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:00.547299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.547267 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:00.565093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.564877 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:00.570195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.569646 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsk8l\" (UniqueName: \"kubernetes.io/projected/bfc85c55-68e7-40d8-b830-eee11c38de6c-kube-api-access-vsk8l\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.571105 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.571059 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:00.591951 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.590703 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:00.599375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.599236 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlnb\" (UniqueName: \"kubernetes.io/projected/096f83eb-1617-4613-b023-864f9cb54f53-kube-api-access-mzlnb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.601826 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.601793 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7sw\" (UniqueName: \"kubernetes.io/projected/6dae9676-856a-44a5-904e-e32d2a114e28-kube-api-access-xl7sw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.611273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.610707 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1aae5a-96d6-43b1-83cb-85d3a097faea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.611273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.610801 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5vt\" (UniqueName: \"kubernetes.io/projected/48312fad-cf56-4728-8f43-ad84bad2537d-kube-api-access-bv5vt\") pod \"maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.611273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.610874 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgprl\" (UniqueName: \"kubernetes.io/projected/0a1aae5a-96d6-43b1-83cb-85d3a097faea-kube-api-access-dgprl\") pod \"maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.611273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.610914 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48312fad-cf56-4728-8f43-ad84bad2537d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.612644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.611772 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48312fad-cf56-4728-8f43-ad84bad2537d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.612644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.612406 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1aae5a-96d6-43b1-83cb-85d3a097faea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.674829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.674750 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5vt\" (UniqueName: \"kubernetes.io/projected/48312fad-cf56-4728-8f43-ad84bad2537d-kube-api-access-bv5vt\") pod \"maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.692707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.692676 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgprl\" (UniqueName: \"kubernetes.io/projected/0a1aae5a-96d6-43b1-83cb-85d3a097faea-kube-api-access-dgprl\") pod \"maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.708679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.708421 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:00.740033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.733916 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:00.744226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.744135 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:00.790341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.790314 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:00.895458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.895244 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:00.897010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.896112 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4"] Nov 02 21:45:00.898709 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:00.898658 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757f3a79_c1cb_46c8_8210_bee23a846f45.slice/crio-81ef9ae41fd58240b546e8d39ab81bec940aa4a3c070aee248979b50a90e5ed0 WatchSource:0}: Error finding container 81ef9ae41fd58240b546e8d39ab81bec940aa4a3c070aee248979b50a90e5ed0: Status 404 returned error can't find the container with id 81ef9ae41fd58240b546e8d39ab81bec940aa4a3c070aee248979b50a90e5ed0 Nov 02 21:45:00.949230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.949015 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:00.992072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:00.992041 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" event={"ID":"757f3a79-c1cb-46c8-8210-bee23a846f45","Type":"ContainerStarted","Data":"81ef9ae41fd58240b546e8d39ab81bec940aa4a3c070aee248979b50a90e5ed0"} Nov 02 21:45:01.308005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.307978 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2"] Nov 02 21:45:01.316195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.316164 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt"] Nov 02 21:45:01.321125 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:01.321090 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16e3380_5275_4fc3_a2c0_e7b45edf8ca2.slice/crio-585c45bd5f4f98d84bd44d402b54c6bc766405e71325fd5869c0f9d5195c4b0c WatchSource:0}: Error finding container 585c45bd5f4f98d84bd44d402b54c6bc766405e71325fd5869c0f9d5195c4b0c: Status 404 returned error can't find the container with id 585c45bd5f4f98d84bd44d402b54c6bc766405e71325fd5869c0f9d5195c4b0c Nov 02 21:45:01.362081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.362009 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7"] Nov 02 21:45:01.366095 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:01.366062 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa2c34b_ff3f_40d6_afad_3172880687f7.slice/crio-891f0d2c33817064ca07e8b1b3dd6ab442b55ae64fb5dc809fcdfd6b0ab7669a WatchSource:0}: Error finding container 891f0d2c33817064ca07e8b1b3dd6ab442b55ae64fb5dc809fcdfd6b0ab7669a: Status 404 returned error can't find the container with id 891f0d2c33817064ca07e8b1b3dd6ab442b55ae64fb5dc809fcdfd6b0ab7669a Nov 02 21:45:01.423947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.423917 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj"] Nov 02 21:45:01.427587 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:01.427555 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5862b7_d0b9_44df_ab25_5c562cae664a.slice/crio-138a659dd0f3703d9e0dbe8965f56f60ea4845f008b3efc21df5f74e9147811d WatchSource:0}: Error finding container 138a659dd0f3703d9e0dbe8965f56f60ea4845f008b3efc21df5f74e9147811d: Status 404 returned error can't find the container with id 138a659dd0f3703d9e0dbe8965f56f60ea4845f008b3efc21df5f74e9147811d Nov 02 21:45:01.438673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.438646 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8"] Nov 02 21:45:01.442558 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:01.442531 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e37857_599f_4d61_9908_0d0b57269498.slice/crio-20352aa901edf67840333a42bb87729c5115b227f167d565b6b7f2b70d252871 WatchSource:0}: Error finding container 20352aa901edf67840333a42bb87729c5115b227f167d565b6b7f2b70d252871: Status 404 returned error can't find the container with id 20352aa901edf67840333a42bb87729c5115b227f167d565b6b7f2b70d252871 Nov 02 21:45:01.465115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.465082 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms"] Nov 02 21:45:01.496984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.496185 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt"] Nov 02 21:45:01.496984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.496273 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7"] Nov 02 21:45:01.565898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.565855 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz"] Nov 02 21:45:01.576769 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:01.576731 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dae9676_856a_44a5_904e_e32d2a114e28.slice/crio-0c0f089e665dfbb74c6db1844288a6193ee14123b8d25d300660f1b7b435be92 WatchSource:0}: Error finding container 0c0f089e665dfbb74c6db1844288a6193ee14123b8d25d300660f1b7b435be92: Status 404 returned error can't find the container with id 0c0f089e665dfbb74c6db1844288a6193ee14123b8d25d300660f1b7b435be92 Nov 02 21:45:01.578284 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:01.578240 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48312fad_cf56_4728_8f43_ad84bad2537d.slice/crio-df0d2b14a75a48f975232caacd1a974113283a339874b02fa93a077edef05fdf WatchSource:0}: Error finding container df0d2b14a75a48f975232caacd1a974113283a339874b02fa93a077edef05fdf: Status 404 returned error can't find the container with id df0d2b14a75a48f975232caacd1a974113283a339874b02fa93a077edef05fdf Nov 02 21:45:01.592985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.592359 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx"] Nov 02 21:45:01.609499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.609472 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh"] Nov 02 21:45:01.721278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.721253 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d"] Nov 02 21:45:01.723412 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:01.723383 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096f83eb_1617_4613_b023_864f9cb54f53.slice/crio-33e3b01474fbff7aee7c031c7da705840510a6b1a4d62a8e6490bd442544a7f1 WatchSource:0}: Error finding container 33e3b01474fbff7aee7c031c7da705840510a6b1a4d62a8e6490bd442544a7f1: Status 404 returned error can't find the container with id 33e3b01474fbff7aee7c031c7da705840510a6b1a4d62a8e6490bd442544a7f1 Nov 02 21:45:01.801419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:01.801388 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:02.013674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.013606 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" event={"ID":"6dae9676-856a-44a5-904e-e32d2a114e28","Type":"ContainerStarted","Data":"0c0f089e665dfbb74c6db1844288a6193ee14123b8d25d300660f1b7b435be92"} Nov 02 21:45:02.019038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.018606 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" event={"ID":"6e5862b7-d0b9-44df-ab25-5c562cae664a","Type":"ContainerStarted","Data":"138a659dd0f3703d9e0dbe8965f56f60ea4845f008b3efc21df5f74e9147811d"} Nov 02 21:45:02.020312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.020288 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" event={"ID":"096f83eb-1617-4613-b023-864f9cb54f53","Type":"ContainerStarted","Data":"33e3b01474fbff7aee7c031c7da705840510a6b1a4d62a8e6490bd442544a7f1"} Nov 02 21:45:02.022180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.022142 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" event={"ID":"d3e37857-599f-4d61-9908-0d0b57269498","Type":"ContainerStarted","Data":"20352aa901edf67840333a42bb87729c5115b227f167d565b6b7f2b70d252871"} Nov 02 21:45:02.024042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.024022 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerStarted","Data":"b94312db21290742c1b7bb916089b21f702b49eccc0058f53c5b08b449d1a69f"} Nov 02 21:45:02.024164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.024152 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerStarted","Data":"891f0d2c33817064ca07e8b1b3dd6ab442b55ae64fb5dc809fcdfd6b0ab7669a"} Nov 02 21:45:02.025862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.025837 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" event={"ID":"0bc41514-68ad-4d9d-bb6c-9ccf26358703","Type":"ContainerStarted","Data":"35fe9a4710e247f537b38772d4ba18c453aab9d42be6678d1e46daa09e153a99"} Nov 02 21:45:02.027303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.027281 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" event={"ID":"0a1aae5a-96d6-43b1-83cb-85d3a097faea","Type":"ContainerStarted","Data":"8b0ac7920bfe8a5f526ee0d631208eedd3417ed782c51b51d6bde0236581f815"} Nov 02 21:45:02.028448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.028428 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" event={"ID":"bfc85c55-68e7-40d8-b830-eee11c38de6c","Type":"ContainerStarted","Data":"114d0c0b245b1d9b40418149f04e5edb3aad00f99743e4fa5cd2a35d44266038"} Nov 02 21:45:02.029929 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.029909 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" event={"ID":"c376f194-53a4-4d9f-bf63-dadcdbf79542","Type":"ContainerStarted","Data":"07c293c3c2ef6592dc366363b065e9f0dc7a25f44b20f7404e1223d90001d4ad"} Nov 02 21:45:02.032384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.032360 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerStarted","Data":"6be6e598c7cc4c3c1fdb764a19da7a1547179d9bf0e85103cf1c733d72415e8c"} Nov 02 21:45:02.032465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.032407 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerStarted","Data":"585c45bd5f4f98d84bd44d402b54c6bc766405e71325fd5869c0f9d5195c4b0c"} Nov 02 21:45:02.034042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.034018 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" event={"ID":"757f3a79-c1cb-46c8-8210-bee23a846f45","Type":"ContainerStarted","Data":"6a1beb7a7a5b7c7a3724d5f376e14b50fc1b0c798e3f719f0c0bf0e149536471"} Nov 02 21:45:02.035235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.035216 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" event={"ID":"48312fad-cf56-4728-8f43-ad84bad2537d","Type":"ContainerStarted","Data":"df0d2b14a75a48f975232caacd1a974113283a339874b02fa93a077edef05fdf"} Nov 02 21:45:02.036503 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.036486 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" event={"ID":"53142d59-d966-4f70-9609-f8613c9b2547","Type":"ContainerStarted","Data":"ee883f16c6591176237d6f69d914e795be37693f3fb4cdcafc9323a6be4842dd"} Nov 02 21:45:02.137556 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.137508 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" podStartSLOduration=2.137493101 podStartE2EDuration="2.137493101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:02.134111787 +0000 UTC m=+1931.689740202" watchObservedRunningTime="2026-11-02 21:45:02.137493101 +0000 UTC m=+1931.693121432" Nov 02 21:45:02.240997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.240922 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" podStartSLOduration=2.240908835 podStartE2EDuration="2.240908835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:02.24014885 +0000 UTC m=+1931.795777190" watchObservedRunningTime="2026-11-02 21:45:02.240908835 +0000 UTC m=+1931.796537219" Nov 02 21:45:02.348118 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.348060 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" podStartSLOduration=2.348042684 podStartE2EDuration="2.348042684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:02.34618421 +0000 UTC m=+1931.901812547" watchObservedRunningTime="2026-11-02 21:45:02.348042684 +0000 UTC m=+1931.903671022" Nov 02 21:45:02.457765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:02.457665 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" podStartSLOduration=2.457647288 podStartE2EDuration="2.457647288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:02.45377083 +0000 UTC m=+1932.009399168" watchObservedRunningTime="2026-11-02 21:45:02.457647288 +0000 UTC m=+1932.013275626" Nov 02 21:45:02.729477 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:02.729405 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16e3380_5275_4fc3_a2c0_e7b45edf8ca2.slice/crio-conmon-6be6e598c7cc4c3c1fdb764a19da7a1547179d9bf0e85103cf1c733d72415e8c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16e3380_5275_4fc3_a2c0_e7b45edf8ca2.slice/crio-6be6e598c7cc4c3c1fdb764a19da7a1547179d9bf0e85103cf1c733d72415e8c.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:45:03.047912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.047882 4441 generic.go:356] "Generic (PLEG): container finished" podID="53142d59-d966-4f70-9609-f8613c9b2547" containerID="93e8d69452db2ed0058aeee3f566f17d03327670c2b7e62fbc17ae7e5cab5812" exitCode=60 Nov 02 21:45:03.048078 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.047982 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" event={"ID":"53142d59-d966-4f70-9609-f8613c9b2547","Type":"ContainerDied","Data":"93e8d69452db2ed0058aeee3f566f17d03327670c2b7e62fbc17ae7e5cab5812"} Nov 02 21:45:03.048259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.048241 4441 scope.go:122] "RemoveContainer" containerID="93e8d69452db2ed0058aeee3f566f17d03327670c2b7e62fbc17ae7e5cab5812" Nov 02 21:45:03.049756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.049733 4441 generic.go:356] "Generic (PLEG): container finished" podID="6dae9676-856a-44a5-904e-e32d2a114e28" containerID="4eec3e90d5503e94d3d4e5c2c78c1088103b6f3b2afd894f349f9d5cbc590961" exitCode=60 Nov 02 21:45:03.049916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.049851 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" event={"ID":"6dae9676-856a-44a5-904e-e32d2a114e28","Type":"ContainerDied","Data":"4eec3e90d5503e94d3d4e5c2c78c1088103b6f3b2afd894f349f9d5cbc590961"} Nov 02 21:45:03.050038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.050025 4441 scope.go:122] "RemoveContainer" containerID="4eec3e90d5503e94d3d4e5c2c78c1088103b6f3b2afd894f349f9d5cbc590961" Nov 02 21:45:03.051845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.051753 4441 generic.go:356] "Generic (PLEG): container finished" podID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerID="2380c3906958943b766946fef8cf41472b8bd3c50813b7fd8e62f622bbace628" exitCode=60 Nov 02 21:45:03.051907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.051841 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" event={"ID":"6e5862b7-d0b9-44df-ab25-5c562cae664a","Type":"ContainerDied","Data":"2380c3906958943b766946fef8cf41472b8bd3c50813b7fd8e62f622bbace628"} Nov 02 21:45:03.052103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.052027 4441 scope.go:122] "RemoveContainer" containerID="2380c3906958943b766946fef8cf41472b8bd3c50813b7fd8e62f622bbace628" Nov 02 21:45:03.055086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.055051 4441 generic.go:356] "Generic (PLEG): container finished" podID="096f83eb-1617-4613-b023-864f9cb54f53" containerID="529cf01dde4ee437bcfe8d1e5e6d1a9e3fb30c778847e3980d66951392087454" exitCode=60 Nov 02 21:45:03.055193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.055079 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" event={"ID":"096f83eb-1617-4613-b023-864f9cb54f53","Type":"ContainerDied","Data":"529cf01dde4ee437bcfe8d1e5e6d1a9e3fb30c778847e3980d66951392087454"} Nov 02 21:45:03.055328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.055312 4441 scope.go:122] "RemoveContainer" containerID="529cf01dde4ee437bcfe8d1e5e6d1a9e3fb30c778847e3980d66951392087454" Nov 02 21:45:03.056863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.056844 4441 generic.go:356] "Generic (PLEG): container finished" podID="d3e37857-599f-4d61-9908-0d0b57269498" containerID="18b4a25e26901e8b0100020076e56c399c758e4fecd692772ff5ee3bf3a62794" exitCode=60 Nov 02 21:45:03.056945 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.056925 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" event={"ID":"d3e37857-599f-4d61-9908-0d0b57269498","Type":"ContainerDied","Data":"18b4a25e26901e8b0100020076e56c399c758e4fecd692772ff5ee3bf3a62794"} Nov 02 21:45:03.057117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.057102 4441 scope.go:122] "RemoveContainer" containerID="18b4a25e26901e8b0100020076e56c399c758e4fecd692772ff5ee3bf3a62794" Nov 02 21:45:03.058511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.058493 4441 generic.go:356] "Generic (PLEG): container finished" podID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerID="b94312db21290742c1b7bb916089b21f702b49eccc0058f53c5b08b449d1a69f" exitCode=60 Nov 02 21:45:03.058598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.058572 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerDied","Data":"b94312db21290742c1b7bb916089b21f702b49eccc0058f53c5b08b449d1a69f"} Nov 02 21:45:03.058779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.058763 4441 scope.go:122] "RemoveContainer" containerID="b94312db21290742c1b7bb916089b21f702b49eccc0058f53c5b08b449d1a69f" Nov 02 21:45:03.060766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.060747 4441 generic.go:356] "Generic (PLEG): container finished" podID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerID="39479f38ce32a8382c2b87f966f23c2386400ebfcb21816c275e707ba0defa8e" exitCode=60 Nov 02 21:45:03.061071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.061057 4441 scope.go:122] "RemoveContainer" containerID="39479f38ce32a8382c2b87f966f23c2386400ebfcb21816c275e707ba0defa8e" Nov 02 21:45:03.061346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.061329 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" event={"ID":"0bc41514-68ad-4d9d-bb6c-9ccf26358703","Type":"ContainerDied","Data":"39479f38ce32a8382c2b87f966f23c2386400ebfcb21816c275e707ba0defa8e"} Nov 02 21:45:03.062837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.062820 4441 generic.go:356] "Generic (PLEG): container finished" podID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerID="693c5af8b25777e4ee15386e0fcb9ff7aede47fdb75a6f20eaaad4d06d1b1b10" exitCode=60 Nov 02 21:45:03.062909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.062876 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" event={"ID":"0a1aae5a-96d6-43b1-83cb-85d3a097faea","Type":"ContainerDied","Data":"693c5af8b25777e4ee15386e0fcb9ff7aede47fdb75a6f20eaaad4d06d1b1b10"} Nov 02 21:45:03.063165 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.063150 4441 scope.go:122] "RemoveContainer" containerID="693c5af8b25777e4ee15386e0fcb9ff7aede47fdb75a6f20eaaad4d06d1b1b10" Nov 02 21:45:03.064620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.064602 4441 generic.go:356] "Generic (PLEG): container finished" podID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerID="e04cb645d2a2a5689ca47956cf2fc60da85613a1f4383a8d020c5a3cfdaeda41" exitCode=60 Nov 02 21:45:03.064703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.064666 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" event={"ID":"bfc85c55-68e7-40d8-b830-eee11c38de6c","Type":"ContainerDied","Data":"e04cb645d2a2a5689ca47956cf2fc60da85613a1f4383a8d020c5a3cfdaeda41"} Nov 02 21:45:03.064920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.064908 4441 scope.go:122] "RemoveContainer" containerID="e04cb645d2a2a5689ca47956cf2fc60da85613a1f4383a8d020c5a3cfdaeda41" Nov 02 21:45:03.066451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.066430 4441 generic.go:356] "Generic (PLEG): container finished" podID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerID="a14a53810f53f788d9274692dc362ac781772da5815258e940b7a3dc258f236e" exitCode=60 Nov 02 21:45:03.066538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.066449 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" event={"ID":"c376f194-53a4-4d9f-bf63-dadcdbf79542","Type":"ContainerDied","Data":"a14a53810f53f788d9274692dc362ac781772da5815258e940b7a3dc258f236e"} Nov 02 21:45:03.066691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.066674 4441 scope.go:122] "RemoveContainer" containerID="a14a53810f53f788d9274692dc362ac781772da5815258e940b7a3dc258f236e" Nov 02 21:45:03.068180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.068160 4441 generic.go:356] "Generic (PLEG): container finished" podID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerID="6be6e598c7cc4c3c1fdb764a19da7a1547179d9bf0e85103cf1c733d72415e8c" exitCode=60 Nov 02 21:45:03.068270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.068252 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerDied","Data":"6be6e598c7cc4c3c1fdb764a19da7a1547179d9bf0e85103cf1c733d72415e8c"} Nov 02 21:45:03.068466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.068447 4441 scope.go:122] "RemoveContainer" containerID="6be6e598c7cc4c3c1fdb764a19da7a1547179d9bf0e85103cf1c733d72415e8c" Nov 02 21:45:03.070090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.070068 4441 generic.go:356] "Generic (PLEG): container finished" podID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerID="6a1beb7a7a5b7c7a3724d5f376e14b50fc1b0c798e3f719f0c0bf0e149536471" exitCode=60 Nov 02 21:45:03.070258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.070097 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" event={"ID":"757f3a79-c1cb-46c8-8210-bee23a846f45","Type":"ContainerDied","Data":"6a1beb7a7a5b7c7a3724d5f376e14b50fc1b0c798e3f719f0c0bf0e149536471"} Nov 02 21:45:03.070415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.070372 4441 scope.go:122] "RemoveContainer" containerID="6a1beb7a7a5b7c7a3724d5f376e14b50fc1b0c798e3f719f0c0bf0e149536471" Nov 02 21:45:03.075934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.075064 4441 generic.go:356] "Generic (PLEG): container finished" podID="48312fad-cf56-4728-8f43-ad84bad2537d" containerID="2e88874e4aca5b5b66baa6fbfebaa9aedc86ace2a389eff76614a39ad3dac98f" exitCode=60 Nov 02 21:45:03.075934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.075112 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" event={"ID":"48312fad-cf56-4728-8f43-ad84bad2537d","Type":"ContainerDied","Data":"2e88874e4aca5b5b66baa6fbfebaa9aedc86ace2a389eff76614a39ad3dac98f"} Nov 02 21:45:03.075934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:03.075413 4441 scope.go:122] "RemoveContainer" containerID="2e88874e4aca5b5b66baa6fbfebaa9aedc86ace2a389eff76614a39ad3dac98f" Nov 02 21:45:04.122876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.122398 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" event={"ID":"0bc41514-68ad-4d9d-bb6c-9ccf26358703","Type":"ContainerStarted","Data":"4307a988418cc52d0a590ef6ab63bcba4247455d2aef961bf47dcd1d6887fcd2"} Nov 02 21:45:04.127752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.126930 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" event={"ID":"0a1aae5a-96d6-43b1-83cb-85d3a097faea","Type":"ContainerStarted","Data":"62b53c472e3b5ab14adbea53c070043bd26945a12022a1cb47f0ad9d8e60e6e9"} Nov 02 21:45:04.129827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.129793 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" event={"ID":"bfc85c55-68e7-40d8-b830-eee11c38de6c","Type":"ContainerStarted","Data":"bf261bbecb881bdc1aa8073a204e22e6e49cbae412bd6d8b0b34439cd1e3ef28"} Nov 02 21:45:04.132539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.132489 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" event={"ID":"c376f194-53a4-4d9f-bf63-dadcdbf79542","Type":"ContainerStarted","Data":"31760334be74820044d7a856ef89ee1cb21216332e26030afd7ce6c3e46e7452"} Nov 02 21:45:04.134310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.134289 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerStarted","Data":"dd14d1161d9138e4379eb882e67623829c7a81b03114afe3b0c59ab09a6f8713"} Nov 02 21:45:04.135984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.135945 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" event={"ID":"757f3a79-c1cb-46c8-8210-bee23a846f45","Type":"ContainerStarted","Data":"095f36c75eeb9af6c58d53d2533f2981e170f3fae29df1b9b9964d287bf4fcae"} Nov 02 21:45:04.137804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.137781 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" event={"ID":"48312fad-cf56-4728-8f43-ad84bad2537d","Type":"ContainerStarted","Data":"d4eb1e36e01f4c8c58df4db4851785927053eb201e3a25ee324e13d4a5b09bb8"} Nov 02 21:45:04.139627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.139603 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" event={"ID":"6e5862b7-d0b9-44df-ab25-5c562cae664a","Type":"ContainerStarted","Data":"66752b6b19f4da5471c1c171015733680e2451caa715434e464e96821836e23e"} Nov 02 21:45:04.141435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.141415 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" event={"ID":"096f83eb-1617-4613-b023-864f9cb54f53","Type":"ContainerStarted","Data":"43c6eaa6f00a1fbb26358888a2913ffc1fb2ef87630ede1d64a355ea22a3069e"} Nov 02 21:45:04.143198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.143180 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerStarted","Data":"b77a513c98810a3b91e219e8f05bac7262b1d854a22842af23c403be6ab807f4"} Nov 02 21:45:04.545712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.545678 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:04.551543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.551499 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" podStartSLOduration=4.551486787 podStartE2EDuration="4.551486787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:04.55070945 +0000 UTC m=+1934.106337787" watchObservedRunningTime="2026-11-02 21:45:04.551486787 +0000 UTC m=+1934.107115121" Nov 02 21:45:04.661879 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.661764 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" podStartSLOduration=4.661746016 podStartE2EDuration="4.661746016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:04.65715965 +0000 UTC m=+1934.212787987" watchObservedRunningTime="2026-11-02 21:45:04.661746016 +0000 UTC m=+1934.217374353" Nov 02 21:45:04.769076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:04.769012 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" podStartSLOduration=4.768992875 podStartE2EDuration="4.768992875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:04.763459985 +0000 UTC m=+1934.319088322" watchObservedRunningTime="2026-11-02 21:45:04.768992875 +0000 UTC m=+1934.324621213" Nov 02 21:45:05.088844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.088789 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" podStartSLOduration=5.088774947 podStartE2EDuration="5.088774947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:05.088316278 +0000 UTC m=+1934.643944628" watchObservedRunningTime="2026-11-02 21:45:05.088774947 +0000 UTC m=+1934.644403323" Nov 02 21:45:05.155093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.155063 4441 generic.go:356] "Generic (PLEG): container finished" podID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerID="4307a988418cc52d0a590ef6ab63bcba4247455d2aef961bf47dcd1d6887fcd2" exitCode=60 Nov 02 21:45:05.155486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.155143 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" event={"ID":"0bc41514-68ad-4d9d-bb6c-9ccf26358703","Type":"ContainerDied","Data":"4307a988418cc52d0a590ef6ab63bcba4247455d2aef961bf47dcd1d6887fcd2"} Nov 02 21:45:05.155486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.155202 4441 scope.go:122] "RemoveContainer" containerID="39479f38ce32a8382c2b87f966f23c2386400ebfcb21816c275e707ba0defa8e" Nov 02 21:45:05.155486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.155475 4441 scope.go:122] "RemoveContainer" containerID="4307a988418cc52d0a590ef6ab63bcba4247455d2aef961bf47dcd1d6887fcd2" Nov 02 21:45:05.155914 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.155869 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt_odh-ai-gateway-infra(0bc41514-68ad-4d9d-bb6c-9ccf26358703)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" Nov 02 21:45:05.157310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.157292 4441 generic.go:356] "Generic (PLEG): container finished" podID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerID="62b53c472e3b5ab14adbea53c070043bd26945a12022a1cb47f0ad9d8e60e6e9" exitCode=60 Nov 02 21:45:05.157428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.157350 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" event={"ID":"0a1aae5a-96d6-43b1-83cb-85d3a097faea","Type":"ContainerDied","Data":"62b53c472e3b5ab14adbea53c070043bd26945a12022a1cb47f0ad9d8e60e6e9"} Nov 02 21:45:05.157620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.157603 4441 scope.go:122] "RemoveContainer" containerID="62b53c472e3b5ab14adbea53c070043bd26945a12022a1cb47f0ad9d8e60e6e9" Nov 02 21:45:05.158028 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.158008 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh_odh-ai-gateway-infra(0a1aae5a-96d6-43b1-83cb-85d3a097faea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" Nov 02 21:45:05.159863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.159838 4441 generic.go:356] "Generic (PLEG): container finished" podID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerID="bf261bbecb881bdc1aa8073a204e22e6e49cbae412bd6d8b0b34439cd1e3ef28" exitCode=60 Nov 02 21:45:05.159955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.159923 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" event={"ID":"bfc85c55-68e7-40d8-b830-eee11c38de6c","Type":"ContainerDied","Data":"bf261bbecb881bdc1aa8073a204e22e6e49cbae412bd6d8b0b34439cd1e3ef28"} Nov 02 21:45:05.160270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.160250 4441 scope.go:122] "RemoveContainer" containerID="bf261bbecb881bdc1aa8073a204e22e6e49cbae412bd6d8b0b34439cd1e3ef28" Nov 02 21:45:05.160628 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.160604 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx_odh-ai-gateway-infra(bfc85c55-68e7-40d8-b830-eee11c38de6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" Nov 02 21:45:05.162074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.162054 4441 generic.go:356] "Generic (PLEG): container finished" podID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerID="31760334be74820044d7a856ef89ee1cb21216332e26030afd7ce6c3e46e7452" exitCode=60 Nov 02 21:45:05.162165 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.162141 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" event={"ID":"c376f194-53a4-4d9f-bf63-dadcdbf79542","Type":"ContainerDied","Data":"31760334be74820044d7a856ef89ee1cb21216332e26030afd7ce6c3e46e7452"} Nov 02 21:45:05.162360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.162343 4441 scope.go:122] "RemoveContainer" containerID="31760334be74820044d7a856ef89ee1cb21216332e26030afd7ce6c3e46e7452" Nov 02 21:45:05.162619 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.162599 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2_odh-ai-gateway-infra(c376f194-53a4-4d9f-bf63-dadcdbf79542)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" Nov 02 21:45:05.164164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.164143 4441 generic.go:356] "Generic (PLEG): container finished" podID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerID="dd14d1161d9138e4379eb882e67623829c7a81b03114afe3b0c59ab09a6f8713" exitCode=60 Nov 02 21:45:05.164252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.164221 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerDied","Data":"dd14d1161d9138e4379eb882e67623829c7a81b03114afe3b0c59ab09a6f8713"} Nov 02 21:45:05.164446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.164429 4441 scope.go:122] "RemoveContainer" containerID="dd14d1161d9138e4379eb882e67623829c7a81b03114afe3b0c59ab09a6f8713" Nov 02 21:45:05.164714 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.164692 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt_odh-ai-gateway-infra(e16e3380-5275-4fc3-a2c0-e7b45edf8ca2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" Nov 02 21:45:05.166292 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.166270 4441 generic.go:356] "Generic (PLEG): container finished" podID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerID="095f36c75eeb9af6c58d53d2533f2981e170f3fae29df1b9b9964d287bf4fcae" exitCode=60 Nov 02 21:45:05.166388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.166309 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" event={"ID":"757f3a79-c1cb-46c8-8210-bee23a846f45","Type":"ContainerDied","Data":"095f36c75eeb9af6c58d53d2533f2981e170f3fae29df1b9b9964d287bf4fcae"} Nov 02 21:45:05.166642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.166525 4441 scope.go:122] "RemoveContainer" containerID="095f36c75eeb9af6c58d53d2533f2981e170f3fae29df1b9b9964d287bf4fcae" Nov 02 21:45:05.166818 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.166795 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4_odh-ai-gateway-infra(757f3a79-c1cb-46c8-8210-bee23a846f45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" Nov 02 21:45:05.168773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.168754 4441 generic.go:356] "Generic (PLEG): container finished" podID="48312fad-cf56-4728-8f43-ad84bad2537d" containerID="d4eb1e36e01f4c8c58df4db4851785927053eb201e3a25ee324e13d4a5b09bb8" exitCode=60 Nov 02 21:45:05.168861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.168829 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" event={"ID":"48312fad-cf56-4728-8f43-ad84bad2537d","Type":"ContainerDied","Data":"d4eb1e36e01f4c8c58df4db4851785927053eb201e3a25ee324e13d4a5b09bb8"} Nov 02 21:45:05.169071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.169053 4441 scope.go:122] "RemoveContainer" containerID="d4eb1e36e01f4c8c58df4db4851785927053eb201e3a25ee324e13d4a5b09bb8" Nov 02 21:45:05.169309 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.169283 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz_odh-ai-gateway-infra(48312fad-cf56-4728-8f43-ad84bad2537d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" Nov 02 21:45:05.170728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.170707 4441 generic.go:356] "Generic (PLEG): container finished" podID="53142d59-d966-4f70-9609-f8613c9b2547" containerID="783dc47e69a3db9896bdd8e1e155185ed8117a69d4289cd79ea937da48f2a828" exitCode=60 Nov 02 21:45:05.170812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.170788 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" event={"ID":"53142d59-d966-4f70-9609-f8613c9b2547","Type":"ContainerDied","Data":"783dc47e69a3db9896bdd8e1e155185ed8117a69d4289cd79ea937da48f2a828"} Nov 02 21:45:05.171241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.170955 4441 scope.go:122] "RemoveContainer" containerID="783dc47e69a3db9896bdd8e1e155185ed8117a69d4289cd79ea937da48f2a828" Nov 02 21:45:05.171241 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.171201 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-ckwms_odh-ai-gateway-infra(53142d59-d966-4f70-9609-f8613c9b2547)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" podUID="53142d59-d966-4f70-9609-f8613c9b2547" Nov 02 21:45:05.172948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.172928 4441 generic.go:356] "Generic (PLEG): container finished" podID="6dae9676-856a-44a5-904e-e32d2a114e28" containerID="a24bb81755c76e43df3b216a2083039051072679dea02b9ff0b73e7d4f624af8" exitCode=60 Nov 02 21:45:05.173067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.173031 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" event={"ID":"6dae9676-856a-44a5-904e-e32d2a114e28","Type":"ContainerDied","Data":"a24bb81755c76e43df3b216a2083039051072679dea02b9ff0b73e7d4f624af8"} Nov 02 21:45:05.173253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.173234 4441 scope.go:122] "RemoveContainer" containerID="a24bb81755c76e43df3b216a2083039051072679dea02b9ff0b73e7d4f624af8" Nov 02 21:45:05.173520 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.173480 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7_odh-ai-gateway-infra(6dae9676-856a-44a5-904e-e32d2a114e28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" Nov 02 21:45:05.175140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.175121 4441 generic.go:356] "Generic (PLEG): container finished" podID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerID="66752b6b19f4da5471c1c171015733680e2451caa715434e464e96821836e23e" exitCode=60 Nov 02 21:45:05.175247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.175209 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" event={"ID":"6e5862b7-d0b9-44df-ab25-5c562cae664a","Type":"ContainerDied","Data":"66752b6b19f4da5471c1c171015733680e2451caa715434e464e96821836e23e"} Nov 02 21:45:05.175416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.175398 4441 scope.go:122] "RemoveContainer" containerID="66752b6b19f4da5471c1c171015733680e2451caa715434e464e96821836e23e" Nov 02 21:45:05.175674 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.175650 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj_odh-ai-gateway-infra(6e5862b7-d0b9-44df-ab25-5c562cae664a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" Nov 02 21:45:05.177270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.177251 4441 generic.go:356] "Generic (PLEG): container finished" podID="096f83eb-1617-4613-b023-864f9cb54f53" containerID="43c6eaa6f00a1fbb26358888a2913ffc1fb2ef87630ede1d64a355ea22a3069e" exitCode=60 Nov 02 21:45:05.177382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.177338 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" event={"ID":"096f83eb-1617-4613-b023-864f9cb54f53","Type":"ContainerDied","Data":"43c6eaa6f00a1fbb26358888a2913ffc1fb2ef87630ede1d64a355ea22a3069e"} Nov 02 21:45:05.177553 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.177537 4441 scope.go:122] "RemoveContainer" containerID="43c6eaa6f00a1fbb26358888a2913ffc1fb2ef87630ede1d64a355ea22a3069e" Nov 02 21:45:05.177805 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.177784 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d_odh-ai-gateway-infra(096f83eb-1617-4613-b023-864f9cb54f53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" podUID="096f83eb-1617-4613-b023-864f9cb54f53" Nov 02 21:45:05.179627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.179273 4441 generic.go:356] "Generic (PLEG): container finished" podID="d3e37857-599f-4d61-9908-0d0b57269498" containerID="aebfd38690ada41e945cd3e5762bb55bc7980f643ec9c43545eb255cd8e7e6ef" exitCode=60 Nov 02 21:45:05.179627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.179386 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" event={"ID":"d3e37857-599f-4d61-9908-0d0b57269498","Type":"ContainerDied","Data":"aebfd38690ada41e945cd3e5762bb55bc7980f643ec9c43545eb255cd8e7e6ef"} Nov 02 21:45:05.179771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.179658 4441 scope.go:122] "RemoveContainer" containerID="aebfd38690ada41e945cd3e5762bb55bc7980f643ec9c43545eb255cd8e7e6ef" Nov 02 21:45:05.180043 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.179958 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8_odh-ai-gateway-infra(d3e37857-599f-4d61-9908-0d0b57269498)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" podUID="d3e37857-599f-4d61-9908-0d0b57269498" Nov 02 21:45:05.180714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.180690 4441 scope.go:122] "RemoveContainer" containerID="693c5af8b25777e4ee15386e0fcb9ff7aede47fdb75a6f20eaaad4d06d1b1b10" Nov 02 21:45:05.182302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.182278 4441 generic.go:356] "Generic (PLEG): container finished" podID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerID="b77a513c98810a3b91e219e8f05bac7262b1d854a22842af23c403be6ab807f4" exitCode=60 Nov 02 21:45:05.182396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.182339 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerDied","Data":"b77a513c98810a3b91e219e8f05bac7262b1d854a22842af23c403be6ab807f4"} Nov 02 21:45:05.182677 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.182654 4441 scope.go:122] "RemoveContainer" containerID="b77a513c98810a3b91e219e8f05bac7262b1d854a22842af23c403be6ab807f4" Nov 02 21:45:05.182889 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:05.182868 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7_odh-ai-gateway-infra(8aa2c34b-ff3f-40d6-afad-3172880687f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" Nov 02 21:45:05.194192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.194159 4441 scope.go:122] "RemoveContainer" containerID="e04cb645d2a2a5689ca47956cf2fc60da85613a1f4383a8d020c5a3cfdaeda41" Nov 02 21:45:05.208218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.208198 4441 scope.go:122] "RemoveContainer" containerID="a14a53810f53f788d9274692dc362ac781772da5815258e940b7a3dc258f236e" Nov 02 21:45:05.220932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.220913 4441 scope.go:122] "RemoveContainer" containerID="6be6e598c7cc4c3c1fdb764a19da7a1547179d9bf0e85103cf1c733d72415e8c" Nov 02 21:45:05.234072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.233921 4441 scope.go:122] "RemoveContainer" containerID="6a1beb7a7a5b7c7a3724d5f376e14b50fc1b0c798e3f719f0c0bf0e149536471" Nov 02 21:45:05.247073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.247055 4441 scope.go:122] "RemoveContainer" containerID="2e88874e4aca5b5b66baa6fbfebaa9aedc86ace2a389eff76614a39ad3dac98f" Nov 02 21:45:05.260267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.260251 4441 scope.go:122] "RemoveContainer" containerID="93e8d69452db2ed0058aeee3f566f17d03327670c2b7e62fbc17ae7e5cab5812" Nov 02 21:45:05.276857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.276836 4441 scope.go:122] "RemoveContainer" containerID="4eec3e90d5503e94d3d4e5c2c78c1088103b6f3b2afd894f349f9d5cbc590961" Nov 02 21:45:05.290784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.290764 4441 scope.go:122] "RemoveContainer" containerID="2380c3906958943b766946fef8cf41472b8bd3c50813b7fd8e62f622bbace628" Nov 02 21:45:05.304050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.303996 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" podStartSLOduration=5.303978311 podStartE2EDuration="5.303978311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:05.301112555 +0000 UTC m=+1934.856740917" watchObservedRunningTime="2026-11-02 21:45:05.303978311 +0000 UTC m=+1934.859606644" Nov 02 21:45:05.304642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.304619 4441 scope.go:122] "RemoveContainer" containerID="529cf01dde4ee437bcfe8d1e5e6d1a9e3fb30c778847e3980d66951392087454" Nov 02 21:45:05.318090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.318072 4441 scope.go:122] "RemoveContainer" containerID="18b4a25e26901e8b0100020076e56c399c758e4fecd692772ff5ee3bf3a62794" Nov 02 21:45:05.330934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.330916 4441 scope.go:122] "RemoveContainer" containerID="b94312db21290742c1b7bb916089b21f702b49eccc0058f53c5b08b449d1a69f" Nov 02 21:45:05.410256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:05.410205 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" podStartSLOduration=5.410191789 podStartE2EDuration="5.410191789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:05.408110459 +0000 UTC m=+1934.963738801" watchObservedRunningTime="2026-11-02 21:45:05.410191789 +0000 UTC m=+1934.965820125" Nov 02 21:45:06.205838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.205806 4441 scope.go:122] "RemoveContainer" containerID="62b53c472e3b5ab14adbea53c070043bd26945a12022a1cb47f0ad9d8e60e6e9" Nov 02 21:45:06.206306 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.206176 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh_odh-ai-gateway-infra(0a1aae5a-96d6-43b1-83cb-85d3a097faea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" Nov 02 21:45:06.207683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.207659 4441 scope.go:122] "RemoveContainer" containerID="bf261bbecb881bdc1aa8073a204e22e6e49cbae412bd6d8b0b34439cd1e3ef28" Nov 02 21:45:06.207943 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.207920 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx_odh-ai-gateway-infra(bfc85c55-68e7-40d8-b830-eee11c38de6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" Nov 02 21:45:06.209305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.209286 4441 scope.go:122] "RemoveContainer" containerID="31760334be74820044d7a856ef89ee1cb21216332e26030afd7ce6c3e46e7452" Nov 02 21:45:06.209549 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.209528 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2_odh-ai-gateway-infra(c376f194-53a4-4d9f-bf63-dadcdbf79542)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" Nov 02 21:45:06.210904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.210882 4441 scope.go:122] "RemoveContainer" containerID="dd14d1161d9138e4379eb882e67623829c7a81b03114afe3b0c59ab09a6f8713" Nov 02 21:45:06.211173 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.211141 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt_odh-ai-gateway-infra(e16e3380-5275-4fc3-a2c0-e7b45edf8ca2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" Nov 02 21:45:06.212429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.212412 4441 scope.go:122] "RemoveContainer" containerID="095f36c75eeb9af6c58d53d2533f2981e170f3fae29df1b9b9964d287bf4fcae" Nov 02 21:45:06.212689 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.212655 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4_odh-ai-gateway-infra(757f3a79-c1cb-46c8-8210-bee23a846f45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" Nov 02 21:45:06.214133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.214106 4441 scope.go:122] "RemoveContainer" containerID="d4eb1e36e01f4c8c58df4db4851785927053eb201e3a25ee324e13d4a5b09bb8" Nov 02 21:45:06.214353 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.214329 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz_odh-ai-gateway-infra(48312fad-cf56-4728-8f43-ad84bad2537d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" Nov 02 21:45:06.215783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.215765 4441 scope.go:122] "RemoveContainer" containerID="783dc47e69a3db9896bdd8e1e155185ed8117a69d4289cd79ea937da48f2a828" Nov 02 21:45:06.216000 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.215964 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-ckwms_odh-ai-gateway-infra(53142d59-d966-4f70-9609-f8613c9b2547)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" podUID="53142d59-d966-4f70-9609-f8613c9b2547" Nov 02 21:45:06.217502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.217486 4441 scope.go:122] "RemoveContainer" containerID="a24bb81755c76e43df3b216a2083039051072679dea02b9ff0b73e7d4f624af8" Nov 02 21:45:06.217718 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.217696 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7_odh-ai-gateway-infra(6dae9676-856a-44a5-904e-e32d2a114e28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" Nov 02 21:45:06.219186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.219166 4441 scope.go:122] "RemoveContainer" containerID="66752b6b19f4da5471c1c171015733680e2451caa715434e464e96821836e23e" Nov 02 21:45:06.219429 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.219408 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj_odh-ai-gateway-infra(6e5862b7-d0b9-44df-ab25-5c562cae664a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" Nov 02 21:45:06.220613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.220593 4441 scope.go:122] "RemoveContainer" containerID="43c6eaa6f00a1fbb26358888a2913ffc1fb2ef87630ede1d64a355ea22a3069e" Nov 02 21:45:06.220850 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.220830 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d_odh-ai-gateway-infra(096f83eb-1617-4613-b023-864f9cb54f53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" podUID="096f83eb-1617-4613-b023-864f9cb54f53" Nov 02 21:45:06.222426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.222408 4441 scope.go:122] "RemoveContainer" containerID="aebfd38690ada41e945cd3e5762bb55bc7980f643ec9c43545eb255cd8e7e6ef" Nov 02 21:45:06.222668 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.222647 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8_odh-ai-gateway-infra(d3e37857-599f-4d61-9908-0d0b57269498)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" podUID="d3e37857-599f-4d61-9908-0d0b57269498" Nov 02 21:45:06.224023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.224006 4441 scope.go:122] "RemoveContainer" containerID="b77a513c98810a3b91e219e8f05bac7262b1d854a22842af23c403be6ab807f4" Nov 02 21:45:06.224237 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.224219 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7_odh-ai-gateway-infra(8aa2c34b-ff3f-40d6-afad-3172880687f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" Nov 02 21:45:06.225761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.225740 4441 scope.go:122] "RemoveContainer" containerID="4307a988418cc52d0a590ef6ab63bcba4247455d2aef961bf47dcd1d6887fcd2" Nov 02 21:45:06.226043 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:06.226019 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt_odh-ai-gateway-infra(0bc41514-68ad-4d9d-bb6c-9ccf26358703)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" Nov 02 21:45:06.368885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:06.368855 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:08.079910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:08.079871 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:08.199636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:08.199608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:10.206902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:10.206861 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:10.619138 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:10.619100 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:45:10.735092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:10.735026 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:10.888641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:10.888543 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:11.013408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:11.013375 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:15.475787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:15.475751 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:15.577188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:15.577157 4441 scope.go:122] "RemoveContainer" containerID="d4eb1e36e01f4c8c58df4db4851785927053eb201e3a25ee324e13d4a5b09bb8" Nov 02 21:45:15.579292 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:15.579264 4441 scope.go:122] "RemoveContainer" containerID="dd14d1161d9138e4379eb882e67623829c7a81b03114afe3b0c59ab09a6f8713" Nov 02 21:45:15.589994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:15.589130 4441 scope.go:122] "RemoveContainer" containerID="4307a988418cc52d0a590ef6ab63bcba4247455d2aef961bf47dcd1d6887fcd2" Nov 02 21:45:15.589994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:15.589566 4441 scope.go:122] "RemoveContainer" containerID="31760334be74820044d7a856ef89ee1cb21216332e26030afd7ce6c3e46e7452" Nov 02 21:45:16.322572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.322526 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" event={"ID":"c376f194-53a4-4d9f-bf63-dadcdbf79542","Type":"ContainerStarted","Data":"7c41b93ed0a97bc20f7a68f0419e9878ef7b86c3b4c3208039a8a0714646a810"} Nov 02 21:45:16.324609 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.324580 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerStarted","Data":"e862fcfc71322fbc63c2ac5b8ec08f2422350b9169569f3ae1d1d3ab03ed9647"} Nov 02 21:45:16.326517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.326495 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" event={"ID":"48312fad-cf56-4728-8f43-ad84bad2537d","Type":"ContainerStarted","Data":"8d79527232cab03bfeaefdaf9e8731c9b1f5faf0998772cd70e2e442007d1b41"} Nov 02 21:45:16.328613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.328591 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" event={"ID":"0bc41514-68ad-4d9d-bb6c-9ccf26358703","Type":"ContainerStarted","Data":"ce002c2f8cf450ade28641f830ae8733e84902befb43e3b410641a97476b5baa"} Nov 02 21:45:16.577215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.577184 4441 scope.go:122] "RemoveContainer" containerID="62b53c472e3b5ab14adbea53c070043bd26945a12022a1cb47f0ad9d8e60e6e9" Nov 02 21:45:16.577665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.577330 4441 scope.go:122] "RemoveContainer" containerID="095f36c75eeb9af6c58d53d2533f2981e170f3fae29df1b9b9964d287bf4fcae" Nov 02 21:45:16.577665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.577534 4441 scope.go:122] "RemoveContainer" containerID="aebfd38690ada41e945cd3e5762bb55bc7980f643ec9c43545eb255cd8e7e6ef" Nov 02 21:45:16.580996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:16.580960 4441 scope.go:122] "RemoveContainer" containerID="66752b6b19f4da5471c1c171015733680e2451caa715434e464e96821836e23e" Nov 02 21:45:17.343113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.343073 4441 generic.go:356] "Generic (PLEG): container finished" podID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerID="ce002c2f8cf450ade28641f830ae8733e84902befb43e3b410641a97476b5baa" exitCode=60 Nov 02 21:45:17.343300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.343136 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" event={"ID":"0bc41514-68ad-4d9d-bb6c-9ccf26358703","Type":"ContainerDied","Data":"ce002c2f8cf450ade28641f830ae8733e84902befb43e3b410641a97476b5baa"} Nov 02 21:45:17.343300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.343198 4441 scope.go:122] "RemoveContainer" containerID="4307a988418cc52d0a590ef6ab63bcba4247455d2aef961bf47dcd1d6887fcd2" Nov 02 21:45:17.343622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.343596 4441 scope.go:122] "RemoveContainer" containerID="ce002c2f8cf450ade28641f830ae8733e84902befb43e3b410641a97476b5baa" Nov 02 21:45:17.344073 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:17.344043 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt_odh-ai-gateway-infra(0bc41514-68ad-4d9d-bb6c-9ccf26358703)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" Nov 02 21:45:17.345645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.345610 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" event={"ID":"0a1aae5a-96d6-43b1-83cb-85d3a097faea","Type":"ContainerStarted","Data":"a3eb76014c3e8732a433f83b76fe96d527a64ebf1878172b5033c1449c335ed9"} Nov 02 21:45:17.350687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.350665 4441 generic.go:356] "Generic (PLEG): container finished" podID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerID="7c41b93ed0a97bc20f7a68f0419e9878ef7b86c3b4c3208039a8a0714646a810" exitCode=60 Nov 02 21:45:17.350797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.350732 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" event={"ID":"c376f194-53a4-4d9f-bf63-dadcdbf79542","Type":"ContainerDied","Data":"7c41b93ed0a97bc20f7a68f0419e9878ef7b86c3b4c3208039a8a0714646a810"} Nov 02 21:45:17.351146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.351129 4441 scope.go:122] "RemoveContainer" containerID="7c41b93ed0a97bc20f7a68f0419e9878ef7b86c3b4c3208039a8a0714646a810" Nov 02 21:45:17.351507 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:17.351480 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2_odh-ai-gateway-infra(c376f194-53a4-4d9f-bf63-dadcdbf79542)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" Nov 02 21:45:17.352824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.352800 4441 generic.go:356] "Generic (PLEG): container finished" podID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerID="e862fcfc71322fbc63c2ac5b8ec08f2422350b9169569f3ae1d1d3ab03ed9647" exitCode=60 Nov 02 21:45:17.352936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.352855 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerDied","Data":"e862fcfc71322fbc63c2ac5b8ec08f2422350b9169569f3ae1d1d3ab03ed9647"} Nov 02 21:45:17.353332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.353313 4441 scope.go:122] "RemoveContainer" containerID="e862fcfc71322fbc63c2ac5b8ec08f2422350b9169569f3ae1d1d3ab03ed9647" Nov 02 21:45:17.353669 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:17.353646 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt_odh-ai-gateway-infra(e16e3380-5275-4fc3-a2c0-e7b45edf8ca2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" Nov 02 21:45:17.355113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.355088 4441 generic.go:356] "Generic (PLEG): container finished" podID="48312fad-cf56-4728-8f43-ad84bad2537d" containerID="8d79527232cab03bfeaefdaf9e8731c9b1f5faf0998772cd70e2e442007d1b41" exitCode=60 Nov 02 21:45:17.355229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.355153 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" event={"ID":"48312fad-cf56-4728-8f43-ad84bad2537d","Type":"ContainerDied","Data":"8d79527232cab03bfeaefdaf9e8731c9b1f5faf0998772cd70e2e442007d1b41"} Nov 02 21:45:17.355464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.355449 4441 scope.go:122] "RemoveContainer" containerID="8d79527232cab03bfeaefdaf9e8731c9b1f5faf0998772cd70e2e442007d1b41" Nov 02 21:45:17.355746 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:17.355722 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz_odh-ai-gateway-infra(48312fad-cf56-4728-8f43-ad84bad2537d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" Nov 02 21:45:17.357264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.357243 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" event={"ID":"6e5862b7-d0b9-44df-ab25-5c562cae664a","Type":"ContainerStarted","Data":"4f9002eff6f6041ff00b056c361adadef8892e2e4f44388b596af3294444226e"} Nov 02 21:45:17.359737 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.359715 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" event={"ID":"d3e37857-599f-4d61-9908-0d0b57269498","Type":"ContainerStarted","Data":"e3697407058c929c9a685bae032be5b653384d2d17a8fee7a5399847525b9c22"} Nov 02 21:45:17.361308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.361293 4441 scope.go:122] "RemoveContainer" containerID="31760334be74820044d7a856ef89ee1cb21216332e26030afd7ce6c3e46e7452" Nov 02 21:45:17.377931 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.377911 4441 scope.go:122] "RemoveContainer" containerID="dd14d1161d9138e4379eb882e67623829c7a81b03114afe3b0c59ab09a6f8713" Nov 02 21:45:17.393922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.393901 4441 scope.go:122] "RemoveContainer" containerID="d4eb1e36e01f4c8c58df4db4851785927053eb201e3a25ee324e13d4a5b09bb8" Nov 02 21:45:17.442434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.442378 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2"] Nov 02 21:45:17.550432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.550398 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt"] Nov 02 21:45:17.576940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.576901 4441 scope.go:122] "RemoveContainer" containerID="43c6eaa6f00a1fbb26358888a2913ffc1fb2ef87630ede1d64a355ea22a3069e" Nov 02 21:45:17.577250 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.577225 4441 scope.go:122] "RemoveContainer" containerID="bf261bbecb881bdc1aa8073a204e22e6e49cbae412bd6d8b0b34439cd1e3ef28" Nov 02 21:45:17.578405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.578386 4441 scope.go:122] "RemoveContainer" containerID="b77a513c98810a3b91e219e8f05bac7262b1d854a22842af23c403be6ab807f4" Nov 02 21:45:17.660492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.660460 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz"] Nov 02 21:45:17.772776 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.772699 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt"] Nov 02 21:45:17.780014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:17.779950 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" podStartSLOduration=17.779936771 podStartE2EDuration="17.779936771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:17.774568235 +0000 UTC m=+1947.330196572" watchObservedRunningTime="2026-11-02 21:45:17.779936771 +0000 UTC m=+1947.335565107" Nov 02 21:45:18.372624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.372521 4441 generic.go:356] "Generic (PLEG): container finished" podID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerID="4f9002eff6f6041ff00b056c361adadef8892e2e4f44388b596af3294444226e" exitCode=60 Nov 02 21:45:18.372780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.372631 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" event={"ID":"6e5862b7-d0b9-44df-ab25-5c562cae664a","Type":"ContainerDied","Data":"4f9002eff6f6041ff00b056c361adadef8892e2e4f44388b596af3294444226e"} Nov 02 21:45:18.372780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.372718 4441 scope.go:122] "RemoveContainer" containerID="66752b6b19f4da5471c1c171015733680e2451caa715434e464e96821836e23e" Nov 02 21:45:18.373240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.373220 4441 scope.go:122] "RemoveContainer" containerID="4f9002eff6f6041ff00b056c361adadef8892e2e4f44388b596af3294444226e" Nov 02 21:45:18.373663 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:18.373638 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj_odh-ai-gateway-infra(6e5862b7-d0b9-44df-ab25-5c562cae664a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" Nov 02 21:45:18.375454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.375429 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" event={"ID":"096f83eb-1617-4613-b023-864f9cb54f53","Type":"ContainerStarted","Data":"5d21e9b6480675ccf1d34facb29512e81ad40f0d92dd3070576358cf80658775"} Nov 02 21:45:18.377615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.377595 4441 generic.go:356] "Generic (PLEG): container finished" podID="d3e37857-599f-4d61-9908-0d0b57269498" containerID="e3697407058c929c9a685bae032be5b653384d2d17a8fee7a5399847525b9c22" exitCode=60 Nov 02 21:45:18.377742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.377650 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" event={"ID":"d3e37857-599f-4d61-9908-0d0b57269498","Type":"ContainerDied","Data":"e3697407058c929c9a685bae032be5b653384d2d17a8fee7a5399847525b9c22"} Nov 02 21:45:18.378057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.378039 4441 scope.go:122] "RemoveContainer" containerID="e3697407058c929c9a685bae032be5b653384d2d17a8fee7a5399847525b9c22" Nov 02 21:45:18.378414 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:18.378395 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8_odh-ai-gateway-infra(d3e37857-599f-4d61-9908-0d0b57269498)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" podUID="d3e37857-599f-4d61-9908-0d0b57269498" Nov 02 21:45:18.379786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.379753 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerStarted","Data":"1e40d9b622252712cb37a155b48abdf80d267ea949c68bcb02f3bdc743e00d95"} Nov 02 21:45:18.383872 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.383851 4441 generic.go:356] "Generic (PLEG): container finished" podID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerID="a3eb76014c3e8732a433f83b76fe96d527a64ebf1878172b5033c1449c335ed9" exitCode=60 Nov 02 21:45:18.383992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.383905 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" event={"ID":"0a1aae5a-96d6-43b1-83cb-85d3a097faea","Type":"ContainerDied","Data":"a3eb76014c3e8732a433f83b76fe96d527a64ebf1878172b5033c1449c335ed9"} Nov 02 21:45:18.384222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.384204 4441 scope.go:122] "RemoveContainer" containerID="a3eb76014c3e8732a433f83b76fe96d527a64ebf1878172b5033c1449c335ed9" Nov 02 21:45:18.384513 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:18.384490 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh_odh-ai-gateway-infra(0a1aae5a-96d6-43b1-83cb-85d3a097faea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" Nov 02 21:45:18.386455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.386433 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" event={"ID":"bfc85c55-68e7-40d8-b830-eee11c38de6c","Type":"ContainerStarted","Data":"249d6d392a4a8bc629a04278bf2819656260f867fc8521633c3f270df7046dc4"} Nov 02 21:45:18.389574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.389553 4441 scope.go:122] "RemoveContainer" containerID="aebfd38690ada41e945cd3e5762bb55bc7980f643ec9c43545eb255cd8e7e6ef" Nov 02 21:45:18.394230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.394201 4441 generic.go:356] "Generic (PLEG): container finished" podID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerID="31e300c676db0d4f9ff621ff444c5fa5fb899452d85a6d3b55dbe4eaf438e492" exitCode=60 Nov 02 21:45:18.394330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.394313 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" event={"ID":"757f3a79-c1cb-46c8-8210-bee23a846f45","Type":"ContainerDied","Data":"31e300c676db0d4f9ff621ff444c5fa5fb899452d85a6d3b55dbe4eaf438e492"} Nov 02 21:45:18.394745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.394723 4441 scope.go:122] "RemoveContainer" containerID="31e300c676db0d4f9ff621ff444c5fa5fb899452d85a6d3b55dbe4eaf438e492" Nov 02 21:45:18.395122 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:18.395100 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4_odh-ai-gateway-infra(757f3a79-c1cb-46c8-8210-bee23a846f45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" Nov 02 21:45:18.408004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.407730 4441 scope.go:122] "RemoveContainer" containerID="62b53c472e3b5ab14adbea53c070043bd26945a12022a1cb47f0ad9d8e60e6e9" Nov 02 21:45:18.428875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.428816 4441 scope.go:122] "RemoveContainer" containerID="095f36c75eeb9af6c58d53d2533f2981e170f3fae29df1b9b9964d287bf4fcae" Nov 02 21:45:18.574636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.574595 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh"] Nov 02 21:45:18.578599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.577660 4441 scope.go:122] "RemoveContainer" containerID="a24bb81755c76e43df3b216a2083039051072679dea02b9ff0b73e7d4f624af8" Nov 02 21:45:18.667687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.667615 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:18.716144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.716112 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc41514-68ad-4d9d-bb6c-9ccf26358703-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc41514-68ad-4d9d-bb6c-9ccf26358703-maas-api-service-ca\") pod \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " Nov 02 21:45:18.716324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.716289 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc41514-68ad-4d9d-bb6c-9ccf26358703-kube-api-access-7f6q4\" (UniqueName: \"kubernetes.io/projected/0bc41514-68ad-4d9d-bb6c-9ccf26358703-kube-api-access-7f6q4\") pod \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\" (UID: \"0bc41514-68ad-4d9d-bb6c-9ccf26358703\") " Nov 02 21:45:18.716550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.716520 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc41514-68ad-4d9d-bb6c-9ccf26358703-maas-api-service-ca" pod "0bc41514-68ad-4d9d-bb6c-9ccf26358703" (UID: "0bc41514-68ad-4d9d-bb6c-9ccf26358703"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:18.716979 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.716944 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc41514-68ad-4d9d-bb6c-9ccf26358703-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:18.770539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.770282 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc41514-68ad-4d9d-bb6c-9ccf26358703-kube-api-access-7f6q4" pod "0bc41514-68ad-4d9d-bb6c-9ccf26358703" (UID: "0bc41514-68ad-4d9d-bb6c-9ccf26358703"). InnerVolumeSpecName "kube-api-access-7f6q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:18.789674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.789645 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8"] Nov 02 21:45:18.818635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.818603 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f6q4\" (UniqueName: \"kubernetes.io/projected/0bc41514-68ad-4d9d-bb6c-9ccf26358703-kube-api-access-7f6q4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:18.992152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.992127 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:18.995558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:18.995535 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:19.000583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.000555 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:19.003657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.003636 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj"] Nov 02 21:45:19.020504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.020479 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-kube-api-access-vv88l\" (UniqueName: \"kubernetes.io/projected/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-kube-api-access-vv88l\") pod \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " Nov 02 21:45:19.020627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.020564 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-maas-api-service-ca\") pod \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\" (UID: \"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2\") " Nov 02 21:45:19.021070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.021038 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-maas-api-service-ca" pod "e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" (UID: "e16e3380-5275-4fc3-a2c0-e7b45edf8ca2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.022682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.022658 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-kube-api-access-vv88l" pod "e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" (UID: "e16e3380-5275-4fc3-a2c0-e7b45edf8ca2"). InnerVolumeSpecName "kube-api-access-vv88l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:19.121498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.121472 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c376f194-53a4-4d9f-bf63-dadcdbf79542-kube-api-access-qhz68\" (UniqueName: \"kubernetes.io/projected/c376f194-53a4-4d9f-bf63-dadcdbf79542-kube-api-access-qhz68\") pod \"c376f194-53a4-4d9f-bf63-dadcdbf79542\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " Nov 02 21:45:19.121594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.121506 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48312fad-cf56-4728-8f43-ad84bad2537d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48312fad-cf56-4728-8f43-ad84bad2537d-maas-api-service-ca\") pod \"48312fad-cf56-4728-8f43-ad84bad2537d\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " Nov 02 21:45:19.121671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.121658 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c376f194-53a4-4d9f-bf63-dadcdbf79542-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c376f194-53a4-4d9f-bf63-dadcdbf79542-maas-api-service-ca\") pod \"c376f194-53a4-4d9f-bf63-dadcdbf79542\" (UID: \"c376f194-53a4-4d9f-bf63-dadcdbf79542\") " Nov 02 21:45:19.121746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.121726 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48312fad-cf56-4728-8f43-ad84bad2537d-kube-api-access-bv5vt\" (UniqueName: \"kubernetes.io/projected/48312fad-cf56-4728-8f43-ad84bad2537d-kube-api-access-bv5vt\") pod \"48312fad-cf56-4728-8f43-ad84bad2537d\" (UID: \"48312fad-cf56-4728-8f43-ad84bad2537d\") " Nov 02 21:45:19.121921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.121870 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48312fad-cf56-4728-8f43-ad84bad2537d-maas-api-service-ca" pod "48312fad-cf56-4728-8f43-ad84bad2537d" (UID: "48312fad-cf56-4728-8f43-ad84bad2537d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.122058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.122034 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c376f194-53a4-4d9f-bf63-dadcdbf79542-maas-api-service-ca" pod "c376f194-53a4-4d9f-bf63-dadcdbf79542" (UID: "c376f194-53a4-4d9f-bf63-dadcdbf79542"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.122343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.122326 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c376f194-53a4-4d9f-bf63-dadcdbf79542-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.122416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.122344 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48312fad-cf56-4728-8f43-ad84bad2537d-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.122416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.122354 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv88l\" (UniqueName: \"kubernetes.io/projected/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-kube-api-access-vv88l\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.122416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.122366 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.123533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.123509 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c376f194-53a4-4d9f-bf63-dadcdbf79542-kube-api-access-qhz68" pod "c376f194-53a4-4d9f-bf63-dadcdbf79542" (UID: "c376f194-53a4-4d9f-bf63-dadcdbf79542"). InnerVolumeSpecName "kube-api-access-qhz68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:19.123615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.123601 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48312fad-cf56-4728-8f43-ad84bad2537d-kube-api-access-bv5vt" pod "48312fad-cf56-4728-8f43-ad84bad2537d" (UID: "48312fad-cf56-4728-8f43-ad84bad2537d"). InnerVolumeSpecName "kube-api-access-bv5vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:19.224087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.224003 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv5vt\" (UniqueName: \"kubernetes.io/projected/48312fad-cf56-4728-8f43-ad84bad2537d-kube-api-access-bv5vt\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.224087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.224039 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhz68\" (UniqueName: \"kubernetes.io/projected/c376f194-53a4-4d9f-bf63-dadcdbf79542-kube-api-access-qhz68\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.407893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.407850 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" event={"ID":"6dae9676-856a-44a5-904e-e32d2a114e28","Type":"ContainerStarted","Data":"347db576b74cab4cb2a346b4abd129a31e87a831ba3ebe87ee5e2edd596533e7"} Nov 02 21:45:19.418161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.418127 4441 generic.go:356] "Generic (PLEG): container finished" podID="096f83eb-1617-4613-b023-864f9cb54f53" containerID="5d21e9b6480675ccf1d34facb29512e81ad40f0d92dd3070576358cf80658775" exitCode=60 Nov 02 21:45:19.418286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.418188 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" event={"ID":"096f83eb-1617-4613-b023-864f9cb54f53","Type":"ContainerDied","Data":"5d21e9b6480675ccf1d34facb29512e81ad40f0d92dd3070576358cf80658775"} Nov 02 21:45:19.418286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.418259 4441 scope.go:122] "RemoveContainer" containerID="43c6eaa6f00a1fbb26358888a2913ffc1fb2ef87630ede1d64a355ea22a3069e" Nov 02 21:45:19.418841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.418744 4441 scope.go:122] "RemoveContainer" containerID="5d21e9b6480675ccf1d34facb29512e81ad40f0d92dd3070576358cf80658775" Nov 02 21:45:19.419238 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:19.419203 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d_odh-ai-gateway-infra(096f83eb-1617-4613-b023-864f9cb54f53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" podUID="096f83eb-1617-4613-b023-864f9cb54f53" Nov 02 21:45:19.424446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.424296 4441 generic.go:356] "Generic (PLEG): container finished" podID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerID="1e40d9b622252712cb37a155b48abdf80d267ea949c68bcb02f3bdc743e00d95" exitCode=60 Nov 02 21:45:19.424446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.424392 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerDied","Data":"1e40d9b622252712cb37a155b48abdf80d267ea949c68bcb02f3bdc743e00d95"} Nov 02 21:45:19.424779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.424757 4441 scope.go:122] "RemoveContainer" containerID="1e40d9b622252712cb37a155b48abdf80d267ea949c68bcb02f3bdc743e00d95" Nov 02 21:45:19.425184 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:19.425157 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7_odh-ai-gateway-infra(8aa2c34b-ff3f-40d6-afad-3172880687f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" Nov 02 21:45:19.426941 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.426913 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" Nov 02 21:45:19.427148 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.427124 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt" event={"ID":"0bc41514-68ad-4d9d-bb6c-9ccf26358703","Type":"ContainerDied","Data":"35fe9a4710e247f537b38772d4ba18c453aab9d42be6678d1e46daa09e153a99"} Nov 02 21:45:19.438307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.437341 4441 generic.go:356] "Generic (PLEG): container finished" podID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerID="249d6d392a4a8bc629a04278bf2819656260f867fc8521633c3f270df7046dc4" exitCode=60 Nov 02 21:45:19.438307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.437443 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" event={"ID":"bfc85c55-68e7-40d8-b830-eee11c38de6c","Type":"ContainerDied","Data":"249d6d392a4a8bc629a04278bf2819656260f867fc8521633c3f270df7046dc4"} Nov 02 21:45:19.438307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.437841 4441 scope.go:122] "RemoveContainer" containerID="249d6d392a4a8bc629a04278bf2819656260f867fc8521633c3f270df7046dc4" Nov 02 21:45:19.438307 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:19.438240 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx_odh-ai-gateway-infra(bfc85c55-68e7-40d8-b830-eee11c38de6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" Nov 02 21:45:19.440623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.440609 4441 scope.go:122] "RemoveContainer" containerID="b77a513c98810a3b91e219e8f05bac7262b1d854a22842af23c403be6ab807f4" Nov 02 21:45:19.440950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.440932 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" Nov 02 21:45:19.441064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.440942 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2" event={"ID":"c376f194-53a4-4d9f-bf63-dadcdbf79542","Type":"ContainerDied","Data":"07c293c3c2ef6592dc366363b065e9f0dc7a25f44b20f7404e1223d90001d4ad"} Nov 02 21:45:19.444470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.444046 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" Nov 02 21:45:19.444470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.444059 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt" event={"ID":"e16e3380-5275-4fc3-a2c0-e7b45edf8ca2","Type":"ContainerDied","Data":"585c45bd5f4f98d84bd44d402b54c6bc766405e71325fd5869c0f9d5195c4b0c"} Nov 02 21:45:19.455034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.447301 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" event={"ID":"48312fad-cf56-4728-8f43-ad84bad2537d","Type":"ContainerDied","Data":"df0d2b14a75a48f975232caacd1a974113283a339874b02fa93a077edef05fdf"} Nov 02 21:45:19.455034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.447385 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz" Nov 02 21:45:19.484225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.484189 4441 scope.go:122] "RemoveContainer" containerID="ce002c2f8cf450ade28641f830ae8733e84902befb43e3b410641a97476b5baa" Nov 02 21:45:19.499707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.499688 4441 scope.go:122] "RemoveContainer" containerID="bf261bbecb881bdc1aa8073a204e22e6e49cbae412bd6d8b0b34439cd1e3ef28" Nov 02 21:45:19.522673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.521380 4441 scope.go:122] "RemoveContainer" containerID="7c41b93ed0a97bc20f7a68f0419e9878ef7b86c3b4c3208039a8a0714646a810" Nov 02 21:45:19.541915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.541895 4441 scope.go:122] "RemoveContainer" containerID="e862fcfc71322fbc63c2ac5b8ec08f2422350b9169569f3ae1d1d3ab03ed9647" Nov 02 21:45:19.561086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.561063 4441 scope.go:122] "RemoveContainer" containerID="8d79527232cab03bfeaefdaf9e8731c9b1f5faf0998772cd70e2e442007d1b41" Nov 02 21:45:19.581290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.580898 4441 scope.go:122] "RemoveContainer" containerID="783dc47e69a3db9896bdd8e1e155185ed8117a69d4289cd79ea937da48f2a828" Nov 02 21:45:19.632956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.632929 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:19.714589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.713978 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7"] Nov 02 21:45:19.732420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.732398 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e37857-599f-4d61-9908-0d0b57269498-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e37857-599f-4d61-9908-0d0b57269498-maas-api-service-ca\") pod \"d3e37857-599f-4d61-9908-0d0b57269498\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " Nov 02 21:45:19.732693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.732679 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e37857-599f-4d61-9908-0d0b57269498-kube-api-access-n2c6d\" (UniqueName: \"kubernetes.io/projected/d3e37857-599f-4d61-9908-0d0b57269498-kube-api-access-n2c6d\") pod \"d3e37857-599f-4d61-9908-0d0b57269498\" (UID: \"d3e37857-599f-4d61-9908-0d0b57269498\") " Nov 02 21:45:19.732808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.732784 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e37857-599f-4d61-9908-0d0b57269498-maas-api-service-ca" pod "d3e37857-599f-4d61-9908-0d0b57269498" (UID: "d3e37857-599f-4d61-9908-0d0b57269498"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.735168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.735102 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e37857-599f-4d61-9908-0d0b57269498-kube-api-access-n2c6d" pod "d3e37857-599f-4d61-9908-0d0b57269498" (UID: "d3e37857-599f-4d61-9908-0d0b57269498"). InnerVolumeSpecName "kube-api-access-n2c6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:19.736311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.736284 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2c6d\" (UniqueName: \"kubernetes.io/projected/d3e37857-599f-4d61-9908-0d0b57269498-kube-api-access-n2c6d\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.736311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.736314 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e37857-599f-4d61-9908-0d0b57269498-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.744473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.744449 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:19.752794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.752776 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:19.837423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.837389 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1aae5a-96d6-43b1-83cb-85d3a097faea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1aae5a-96d6-43b1-83cb-85d3a097faea-maas-api-service-ca\") pod \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " Nov 02 21:45:19.837558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.837494 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5862b7-d0b9-44df-ab25-5c562cae664a-kube-api-access-2cm69\" (UniqueName: \"kubernetes.io/projected/6e5862b7-d0b9-44df-ab25-5c562cae664a-kube-api-access-2cm69\") pod \"6e5862b7-d0b9-44df-ab25-5c562cae664a\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " Nov 02 21:45:19.837632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.837597 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1aae5a-96d6-43b1-83cb-85d3a097faea-kube-api-access-dgprl\" (UniqueName: \"kubernetes.io/projected/0a1aae5a-96d6-43b1-83cb-85d3a097faea-kube-api-access-dgprl\") pod \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\" (UID: \"0a1aae5a-96d6-43b1-83cb-85d3a097faea\") " Nov 02 21:45:19.837690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.837642 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5862b7-d0b9-44df-ab25-5c562cae664a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5862b7-d0b9-44df-ab25-5c562cae664a-maas-api-service-ca\") pod \"6e5862b7-d0b9-44df-ab25-5c562cae664a\" (UID: \"6e5862b7-d0b9-44df-ab25-5c562cae664a\") " Nov 02 21:45:19.837778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.837754 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1aae5a-96d6-43b1-83cb-85d3a097faea-maas-api-service-ca" pod "0a1aae5a-96d6-43b1-83cb-85d3a097faea" (UID: "0a1aae5a-96d6-43b1-83cb-85d3a097faea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.838008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.837983 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5862b7-d0b9-44df-ab25-5c562cae664a-maas-api-service-ca" pod "6e5862b7-d0b9-44df-ab25-5c562cae664a" (UID: "6e5862b7-d0b9-44df-ab25-5c562cae664a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.838146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.838133 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1aae5a-96d6-43b1-83cb-85d3a097faea-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.838203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.838157 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5862b7-d0b9-44df-ab25-5c562cae664a-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.839565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.839546 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5862b7-d0b9-44df-ab25-5c562cae664a-kube-api-access-2cm69" pod "6e5862b7-d0b9-44df-ab25-5c562cae664a" (UID: "6e5862b7-d0b9-44df-ab25-5c562cae664a"). InnerVolumeSpecName "kube-api-access-2cm69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:19.839610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.839588 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1aae5a-96d6-43b1-83cb-85d3a097faea-kube-api-access-dgprl" pod "0a1aae5a-96d6-43b1-83cb-85d3a097faea" (UID: "0a1aae5a-96d6-43b1-83cb-85d3a097faea"). InnerVolumeSpecName "kube-api-access-dgprl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:19.939947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.939912 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cm69\" (UniqueName: \"kubernetes.io/projected/6e5862b7-d0b9-44df-ab25-5c562cae664a-kube-api-access-2cm69\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.940093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.939952 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgprl\" (UniqueName: \"kubernetes.io/projected/0a1aae5a-96d6-43b1-83cb-85d3a097faea-kube-api-access-dgprl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.955259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.955229 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt"] Nov 02 21:45:19.960718 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:19.960695 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a1bf7d-29894265-sj4xt"] Nov 02 21:45:20.008698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.008643 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:20.031935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.031909 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx"] Nov 02 21:45:20.172944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.172910 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz"] Nov 02 21:45:20.177393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.177368 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b983e02-29894265-kv9hz"] Nov 02 21:45:20.185328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.185300 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d"] Nov 02 21:45:20.247922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.247888 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4"] Nov 02 21:45:20.256254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.256207 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" podStartSLOduration=20.256194527 podStartE2EDuration="20.256194527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:20.255226741 +0000 UTC m=+1949.810855078" watchObservedRunningTime="2026-11-02 21:45:20.256194527 +0000 UTC m=+1949.811822861" Nov 02 21:45:20.414461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.414435 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:20.457721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.457691 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" event={"ID":"53142d59-d966-4f70-9609-f8613c9b2547","Type":"ContainerStarted","Data":"3db385da4c657604c6de8c0d2a062f297d0926f9baa441b8429ac0ea1c4005a5"} Nov 02 21:45:20.460159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.460132 4441 generic.go:356] "Generic (PLEG): container finished" podID="6dae9676-856a-44a5-904e-e32d2a114e28" containerID="347db576b74cab4cb2a346b4abd129a31e87a831ba3ebe87ee5e2edd596533e7" exitCode=60 Nov 02 21:45:20.460360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.460164 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" event={"ID":"6dae9676-856a-44a5-904e-e32d2a114e28","Type":"ContainerDied","Data":"347db576b74cab4cb2a346b4abd129a31e87a831ba3ebe87ee5e2edd596533e7"} Nov 02 21:45:20.460360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.460224 4441 scope.go:122] "RemoveContainer" containerID="a24bb81755c76e43df3b216a2083039051072679dea02b9ff0b73e7d4f624af8" Nov 02 21:45:20.461419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.461399 4441 scope.go:122] "RemoveContainer" containerID="347db576b74cab4cb2a346b4abd129a31e87a831ba3ebe87ee5e2edd596533e7" Nov 02 21:45:20.461848 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:20.461804 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7_odh-ai-gateway-infra(6dae9676-856a-44a5-904e-e32d2a114e28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" Nov 02 21:45:20.462178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.462154 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" event={"ID":"6e5862b7-d0b9-44df-ab25-5c562cae664a","Type":"ContainerDied","Data":"138a659dd0f3703d9e0dbe8965f56f60ea4845f008b3efc21df5f74e9147811d"} Nov 02 21:45:20.462297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.462173 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj" Nov 02 21:45:20.467512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.467489 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" event={"ID":"d3e37857-599f-4d61-9908-0d0b57269498","Type":"ContainerDied","Data":"20352aa901edf67840333a42bb87729c5115b227f167d565b6b7f2b70d252871"} Nov 02 21:45:20.467637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.467576 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8" Nov 02 21:45:20.470701 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.470676 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757f3a79-c1cb-46c8-8210-bee23a846f45-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757f3a79-c1cb-46c8-8210-bee23a846f45-maas-api-service-ca\") pod \"757f3a79-c1cb-46c8-8210-bee23a846f45\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " Nov 02 21:45:20.470896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.470880 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757f3a79-c1cb-46c8-8210-bee23a846f45-kube-api-access-bjprj\" (UniqueName: \"kubernetes.io/projected/757f3a79-c1cb-46c8-8210-bee23a846f45-kube-api-access-bjprj\") pod \"757f3a79-c1cb-46c8-8210-bee23a846f45\" (UID: \"757f3a79-c1cb-46c8-8210-bee23a846f45\") " Nov 02 21:45:20.472050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.471935 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757f3a79-c1cb-46c8-8210-bee23a846f45-maas-api-service-ca" pod "757f3a79-c1cb-46c8-8210-bee23a846f45" (UID: "757f3a79-c1cb-46c8-8210-bee23a846f45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:20.479771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.479753 4441 scope.go:122] "RemoveContainer" containerID="4f9002eff6f6041ff00b056c361adadef8892e2e4f44388b596af3294444226e" Nov 02 21:45:20.479866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.479808 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757f3a79-c1cb-46c8-8210-bee23a846f45-kube-api-access-bjprj" pod "757f3a79-c1cb-46c8-8210-bee23a846f45" (UID: "757f3a79-c1cb-46c8-8210-bee23a846f45"). InnerVolumeSpecName "kube-api-access-bjprj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:20.491521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.491491 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" event={"ID":"0a1aae5a-96d6-43b1-83cb-85d3a097faea","Type":"ContainerDied","Data":"8b0ac7920bfe8a5f526ee0d631208eedd3417ed782c51b51d6bde0236581f815"} Nov 02 21:45:20.491625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.491581 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh" Nov 02 21:45:20.508306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.508229 4441 scope.go:122] "RemoveContainer" containerID="e3697407058c929c9a685bae032be5b653384d2d17a8fee7a5399847525b9c22" Nov 02 21:45:20.519002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.517563 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" event={"ID":"757f3a79-c1cb-46c8-8210-bee23a846f45","Type":"ContainerDied","Data":"81ef9ae41fd58240b546e8d39ab81bec940aa4a3c070aee248979b50a90e5ed0"} Nov 02 21:45:20.519002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.517689 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4" Nov 02 21:45:20.548838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.548812 4441 scope.go:122] "RemoveContainer" containerID="a3eb76014c3e8732a433f83b76fe96d527a64ebf1878172b5033c1449c335ed9" Nov 02 21:45:20.561160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.561091 4441 scope.go:122] "RemoveContainer" containerID="31e300c676db0d4f9ff621ff444c5fa5fb899452d85a6d3b55dbe4eaf438e492" Nov 02 21:45:20.574098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.573608 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjprj\" (UniqueName: \"kubernetes.io/projected/757f3a79-c1cb-46c8-8210-bee23a846f45-kube-api-access-bjprj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.574098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.573642 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/757f3a79-c1cb-46c8-8210-bee23a846f45-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.592516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.592492 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" path="/var/lib/kubelet/pods/0bc41514-68ad-4d9d-bb6c-9ccf26358703/volumes" Nov 02 21:45:20.593516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.593496 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" path="/var/lib/kubelet/pods/48312fad-cf56-4728-8f43-ad84bad2537d/volumes" Nov 02 21:45:20.595040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.595018 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2"] Nov 02 21:45:20.602011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.601982 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e2b27-29894265-7trg2"] Nov 02 21:45:20.804091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.804068 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:20.809117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.809096 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:20.812831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.812784 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:20.879340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.878441 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa2c34b-ff3f-40d6-afad-3172880687f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c34b-ff3f-40d6-afad-3172880687f7-maas-api-service-ca\") pod \"8aa2c34b-ff3f-40d6-afad-3172880687f7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " Nov 02 21:45:20.879340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.878678 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc85c55-68e7-40d8-b830-eee11c38de6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc85c55-68e7-40d8-b830-eee11c38de6c-maas-api-service-ca\") pod \"bfc85c55-68e7-40d8-b830-eee11c38de6c\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " Nov 02 21:45:20.879340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.878764 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa2c34b-ff3f-40d6-afad-3172880687f7-kube-api-access-znrmn\" (UniqueName: \"kubernetes.io/projected/8aa2c34b-ff3f-40d6-afad-3172880687f7-kube-api-access-znrmn\") pod \"8aa2c34b-ff3f-40d6-afad-3172880687f7\" (UID: \"8aa2c34b-ff3f-40d6-afad-3172880687f7\") " Nov 02 21:45:20.879340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.878825 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc85c55-68e7-40d8-b830-eee11c38de6c-kube-api-access-vsk8l\" (UniqueName: \"kubernetes.io/projected/bfc85c55-68e7-40d8-b830-eee11c38de6c-kube-api-access-vsk8l\") pod \"bfc85c55-68e7-40d8-b830-eee11c38de6c\" (UID: \"bfc85c55-68e7-40d8-b830-eee11c38de6c\") " Nov 02 21:45:20.879340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.878905 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096f83eb-1617-4613-b023-864f9cb54f53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096f83eb-1617-4613-b023-864f9cb54f53-maas-api-service-ca\") pod \"096f83eb-1617-4613-b023-864f9cb54f53\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " Nov 02 21:45:20.879786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.878927 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa2c34b-ff3f-40d6-afad-3172880687f7-maas-api-service-ca" pod "8aa2c34b-ff3f-40d6-afad-3172880687f7" (UID: "8aa2c34b-ff3f-40d6-afad-3172880687f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:20.879786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.879164 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc85c55-68e7-40d8-b830-eee11c38de6c-maas-api-service-ca" pod "bfc85c55-68e7-40d8-b830-eee11c38de6c" (UID: "bfc85c55-68e7-40d8-b830-eee11c38de6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:20.879786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.879473 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096f83eb-1617-4613-b023-864f9cb54f53-maas-api-service-ca" pod "096f83eb-1617-4613-b023-864f9cb54f53" (UID: "096f83eb-1617-4613-b023-864f9cb54f53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:20.879786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.878994 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096f83eb-1617-4613-b023-864f9cb54f53-kube-api-access-mzlnb\" (UniqueName: \"kubernetes.io/projected/096f83eb-1617-4613-b023-864f9cb54f53-kube-api-access-mzlnb\") pod \"096f83eb-1617-4613-b023-864f9cb54f53\" (UID: \"096f83eb-1617-4613-b023-864f9cb54f53\") " Nov 02 21:45:20.880296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.880248 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c34b-ff3f-40d6-afad-3172880687f7-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.880296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.880273 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096f83eb-1617-4613-b023-864f9cb54f53-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.882405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.882380 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096f83eb-1617-4613-b023-864f9cb54f53-kube-api-access-mzlnb" pod "096f83eb-1617-4613-b023-864f9cb54f53" (UID: "096f83eb-1617-4613-b023-864f9cb54f53"). InnerVolumeSpecName "kube-api-access-mzlnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:20.882934 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.882907 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa2c34b-ff3f-40d6-afad-3172880687f7-kube-api-access-znrmn" pod "8aa2c34b-ff3f-40d6-afad-3172880687f7" (UID: "8aa2c34b-ff3f-40d6-afad-3172880687f7"). InnerVolumeSpecName "kube-api-access-znrmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:20.885022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.884993 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc85c55-68e7-40d8-b830-eee11c38de6c-kube-api-access-vsk8l" pod "bfc85c55-68e7-40d8-b830-eee11c38de6c" (UID: "bfc85c55-68e7-40d8-b830-eee11c38de6c"). InnerVolumeSpecName "kube-api-access-vsk8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:20.922589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.922556 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt"] Nov 02 21:45:20.927892 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.927870 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b7107c-29894265-66fqt"] Nov 02 21:45:20.981577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.981550 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc85c55-68e7-40d8-b830-eee11c38de6c-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.981577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.981575 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrmn\" (UniqueName: \"kubernetes.io/projected/8aa2c34b-ff3f-40d6-afad-3172880687f7-kube-api-access-znrmn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.981712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.981594 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsk8l\" (UniqueName: \"kubernetes.io/projected/bfc85c55-68e7-40d8-b830-eee11c38de6c-kube-api-access-vsk8l\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.981712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:20.981607 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlnb\" (UniqueName: \"kubernetes.io/projected/096f83eb-1617-4613-b023-864f9cb54f53-kube-api-access-mzlnb\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:21.196885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.196791 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh"] Nov 02 21:45:21.201804 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.201773 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d3b375-29894265-rn6hh"] Nov 02 21:45:21.268411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.268383 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7"] Nov 02 21:45:21.545312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.545227 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" event={"ID":"096f83eb-1617-4613-b023-864f9cb54f53","Type":"ContainerDied","Data":"33e3b01474fbff7aee7c031c7da705840510a6b1a4d62a8e6490bd442544a7f1"} Nov 02 21:45:21.545312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.545284 4441 scope.go:122] "RemoveContainer" containerID="5d21e9b6480675ccf1d34facb29512e81ad40f0d92dd3070576358cf80658775" Nov 02 21:45:21.545548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.545377 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d" Nov 02 21:45:21.552731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.552699 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" event={"ID":"8aa2c34b-ff3f-40d6-afad-3172880687f7","Type":"ContainerDied","Data":"891f0d2c33817064ca07e8b1b3dd6ab442b55ae64fb5dc809fcdfd6b0ab7669a"} Nov 02 21:45:21.552866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.552822 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7" Nov 02 21:45:21.559519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.559439 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" Nov 02 21:45:21.559519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.559459 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx" event={"ID":"bfc85c55-68e7-40d8-b830-eee11c38de6c","Type":"ContainerDied","Data":"114d0c0b245b1d9b40418149f04e5edb3aad00f99743e4fa5cd2a35d44266038"} Nov 02 21:45:21.563330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.563302 4441 generic.go:356] "Generic (PLEG): container finished" podID="53142d59-d966-4f70-9609-f8613c9b2547" containerID="3db385da4c657604c6de8c0d2a062f297d0926f9baa441b8429ac0ea1c4005a5" exitCode=60 Nov 02 21:45:21.563560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.563373 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" event={"ID":"53142d59-d966-4f70-9609-f8613c9b2547","Type":"ContainerDied","Data":"3db385da4c657604c6de8c0d2a062f297d0926f9baa441b8429ac0ea1c4005a5"} Nov 02 21:45:21.564024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.564004 4441 scope.go:122] "RemoveContainer" containerID="3db385da4c657604c6de8c0d2a062f297d0926f9baa441b8429ac0ea1c4005a5" Nov 02 21:45:21.564477 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:21.564341 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-ckwms_odh-ai-gateway-infra(53142d59-d966-4f70-9609-f8613c9b2547)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" podUID="53142d59-d966-4f70-9609-f8613c9b2547" Nov 02 21:45:21.570499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.569607 4441 scope.go:122] "RemoveContainer" containerID="1e40d9b622252712cb37a155b48abdf80d267ea949c68bcb02f3bdc743e00d95" Nov 02 21:45:21.577036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.576416 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4"] Nov 02 21:45:21.585386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.583129 4441 scope.go:122] "RemoveContainer" containerID="249d6d392a4a8bc629a04278bf2819656260f867fc8521633c3f270df7046dc4" Nov 02 21:45:21.588859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.588832 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-06366c-29894265-4pxh4"] Nov 02 21:45:21.600843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.600823 4441 scope.go:122] "RemoveContainer" containerID="783dc47e69a3db9896bdd8e1e155185ed8117a69d4289cd79ea937da48f2a828" Nov 02 21:45:21.739243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.739210 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8"] Nov 02 21:45:21.745988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.745600 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9f77b7-29894265-8hhr8"] Nov 02 21:45:21.807517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.807499 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:21.896054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.896024 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dae9676-856a-44a5-904e-e32d2a114e28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dae9676-856a-44a5-904e-e32d2a114e28-maas-api-service-ca\") pod \"6dae9676-856a-44a5-904e-e32d2a114e28\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " Nov 02 21:45:21.896246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.896066 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dae9676-856a-44a5-904e-e32d2a114e28-kube-api-access-xl7sw\" (UniqueName: \"kubernetes.io/projected/6dae9676-856a-44a5-904e-e32d2a114e28-kube-api-access-xl7sw\") pod \"6dae9676-856a-44a5-904e-e32d2a114e28\" (UID: \"6dae9676-856a-44a5-904e-e32d2a114e28\") " Nov 02 21:45:21.896482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.896458 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dae9676-856a-44a5-904e-e32d2a114e28-maas-api-service-ca" pod "6dae9676-856a-44a5-904e-e32d2a114e28" (UID: "6dae9676-856a-44a5-904e-e32d2a114e28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:21.896707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.896679 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dae9676-856a-44a5-904e-e32d2a114e28-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:21.898300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.898275 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dae9676-856a-44a5-904e-e32d2a114e28-kube-api-access-xl7sw" pod "6dae9676-856a-44a5-904e-e32d2a114e28" (UID: "6dae9676-856a-44a5-904e-e32d2a114e28"). InnerVolumeSpecName "kube-api-access-xl7sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:21.952304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.952243 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" podStartSLOduration=21.952228689000002 podStartE2EDuration="21.952228689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:21.947625387 +0000 UTC m=+1951.503253745" watchObservedRunningTime="2026-11-02 21:45:21.952228689 +0000 UTC m=+1951.507857036" Nov 02 21:45:21.998015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:21.997980 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7sw\" (UniqueName: \"kubernetes.io/projected/6dae9676-856a-44a5-904e-e32d2a114e28-kube-api-access-xl7sw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:22.118567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.118470 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj"] Nov 02 21:45:22.123152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.123125 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4d045-29894265-l75jj"] Nov 02 21:45:22.337438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.337397 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7"] Nov 02 21:45:22.341816 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.341790 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894265-24bs7"] Nov 02 21:45:22.444345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.444276 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx"] Nov 02 21:45:22.448763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.448739 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894265-r4qpx"] Nov 02 21:45:22.550403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.550377 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d"] Nov 02 21:45:22.554720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.554697 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f22129-29894265-qh65d"] Nov 02 21:45:22.593544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.593514 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" Nov 02 21:45:22.595718 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.595659 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096f83eb-1617-4613-b023-864f9cb54f53" path="/var/lib/kubelet/pods/096f83eb-1617-4613-b023-864f9cb54f53/volumes" Nov 02 21:45:22.596710 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.596687 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" path="/var/lib/kubelet/pods/0a1aae5a-96d6-43b1-83cb-85d3a097faea/volumes" Nov 02 21:45:22.597921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.597900 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" path="/var/lib/kubelet/pods/6e5862b7-d0b9-44df-ab25-5c562cae664a/volumes" Nov 02 21:45:22.598616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.598593 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" path="/var/lib/kubelet/pods/757f3a79-c1cb-46c8-8210-bee23a846f45/volumes" Nov 02 21:45:22.599458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.599438 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" path="/var/lib/kubelet/pods/8aa2c34b-ff3f-40d6-afad-3172880687f7/volumes" Nov 02 21:45:22.599897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.599881 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" path="/var/lib/kubelet/pods/bfc85c55-68e7-40d8-b830-eee11c38de6c/volumes" Nov 02 21:45:22.600628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.600611 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" path="/var/lib/kubelet/pods/c376f194-53a4-4d9f-bf63-dadcdbf79542/volumes" Nov 02 21:45:22.601089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.601067 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e37857-599f-4d61-9908-0d0b57269498" path="/var/lib/kubelet/pods/d3e37857-599f-4d61-9908-0d0b57269498/volumes" Nov 02 21:45:22.601636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.601617 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" path="/var/lib/kubelet/pods/e16e3380-5275-4fc3-a2c0-e7b45edf8ca2/volumes" Nov 02 21:45:22.602210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.602187 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7" event={"ID":"6dae9676-856a-44a5-904e-e32d2a114e28","Type":"ContainerDied","Data":"0c0f089e665dfbb74c6db1844288a6193ee14123b8d25d300660f1b7b435be92"} Nov 02 21:45:22.602275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.602235 4441 scope.go:122] "RemoveContainer" containerID="347db576b74cab4cb2a346b4abd129a31e87a831ba3ebe87ee5e2edd596533e7" Nov 02 21:45:22.871500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.871469 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7"] Nov 02 21:45:22.877876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.877850 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f13b18-29894265-d94b7"] Nov 02 21:45:22.960747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:22.960718 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms"] Nov 02 21:45:23.132318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.132256 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:23.215627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.215590 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53142d59-d966-4f70-9609-f8613c9b2547-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53142d59-d966-4f70-9609-f8613c9b2547-maas-api-service-ca\") pod \"53142d59-d966-4f70-9609-f8613c9b2547\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " Nov 02 21:45:23.215791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.215701 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53142d59-d966-4f70-9609-f8613c9b2547-kube-api-access-7mp5n\" (UniqueName: \"kubernetes.io/projected/53142d59-d966-4f70-9609-f8613c9b2547-kube-api-access-7mp5n\") pod \"53142d59-d966-4f70-9609-f8613c9b2547\" (UID: \"53142d59-d966-4f70-9609-f8613c9b2547\") " Nov 02 21:45:23.215946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.215923 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53142d59-d966-4f70-9609-f8613c9b2547-maas-api-service-ca" pod "53142d59-d966-4f70-9609-f8613c9b2547" (UID: "53142d59-d966-4f70-9609-f8613c9b2547"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:23.216311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.216290 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53142d59-d966-4f70-9609-f8613c9b2547-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:23.218336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.218296 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53142d59-d966-4f70-9609-f8613c9b2547-kube-api-access-7mp5n" pod "53142d59-d966-4f70-9609-f8613c9b2547" (UID: "53142d59-d966-4f70-9609-f8613c9b2547"). InnerVolumeSpecName "kube-api-access-7mp5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:23.317727 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.317696 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mp5n\" (UniqueName: \"kubernetes.io/projected/53142d59-d966-4f70-9609-f8613c9b2547-kube-api-access-7mp5n\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:23.612274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.612245 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" Nov 02 21:45:23.612726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.612284 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms" event={"ID":"53142d59-d966-4f70-9609-f8613c9b2547","Type":"ContainerDied","Data":"ee883f16c6591176237d6f69d914e795be37693f3fb4cdcafc9323a6be4842dd"} Nov 02 21:45:23.612726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.612348 4441 scope.go:122] "RemoveContainer" containerID="3db385da4c657604c6de8c0d2a062f297d0926f9baa441b8429ac0ea1c4005a5" Nov 02 21:45:23.769400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.769323 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms"] Nov 02 21:45:23.776739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:23.776710 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-ckwms"] Nov 02 21:45:24.447728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:24.447680 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:24.592700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:24.592663 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53142d59-d966-4f70-9609-f8613c9b2547" path="/var/lib/kubelet/pods/53142d59-d966-4f70-9609-f8613c9b2547/volumes" Nov 02 21:45:24.593431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:24.593407 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" path="/var/lib/kubelet/pods/6dae9676-856a-44a5-904e-e32d2a114e28/volumes" Nov 02 21:45:24.648273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:24.648239 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:24.775466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:24.775363 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:29.186925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:29.186884 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:33.699175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:33.699137 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:37.911247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:37.907381 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd"] Nov 02 21:45:37.911247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:37.907647 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" containerID="cri-o://a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69" gracePeriod=30 Nov 02 21:45:38.841410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:38.841256 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6"] Nov 02 21:45:38.841606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:38.841529 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" containerID="cri-o://ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8" gracePeriod=30 Nov 02 21:45:38.860471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:38.860440 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc"] Nov 02 21:45:38.860674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:38.860649 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" containerID="cri-o://b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e" gracePeriod=30 Nov 02 21:45:39.097084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.094921 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:39.250799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.250761 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb"] Nov 02 21:45:39.252208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.252167 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:39.252411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.252395 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:39.252533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.252519 4441 cpu_manager.go:414] "containerMap: removing container" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:39.252697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.252681 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:39.252834 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.252819 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:39.252951 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.252937 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:39.253083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253069 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:39.253191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253177 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:39.253314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253301 4441 cpu_manager.go:414] "containerMap: removing container" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.253420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253398 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.253542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253519 4441 cpu_manager.go:414] "containerMap: removing container" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.253658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253635 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.253778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253756 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:39.253887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.253865 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:39.254023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.254008 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.254125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.254112 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.254257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.254242 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:39.254449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.254434 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:39.254560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.254548 4441 cpu_manager.go:414] "containerMap: removing container" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.254675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.254661 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.254789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.254776 4441 cpu_manager.go:414] "containerMap: removing container" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:39.255019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255000 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:39.255158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255136 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.255158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255153 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255180 4441 cpu_manager.go:414] "containerMap: removing container" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255198 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255208 4441 cpu_manager.go:414] "containerMap: removing container" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255216 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255234 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255241 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255250 4441 cpu_manager.go:414] "containerMap: removing container" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255257 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255265 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255269 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255279 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255284 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255290 4441 cpu_manager.go:414] "containerMap: removing container" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255295 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255310 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255317 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255327 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255332 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255338 4441 cpu_manager.go:414] "containerMap: removing container" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255343 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255349 4441 cpu_manager.go:414] "containerMap: removing container" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255354 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255360 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255365 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:39.255354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255372 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255377 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255384 4441 cpu_manager.go:414] "containerMap: removing container" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255391 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255405 4441 cpu_manager.go:414] "containerMap: removing container" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255411 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255417 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255421 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255432 4441 cpu_manager.go:414] "containerMap: removing container" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255439 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255450 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255461 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255478 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255485 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255518 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255529 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255552 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255564 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255761 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255775 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255783 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255794 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255803 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255812 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255822 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255830 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255839 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255848 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255859 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255869 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255877 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255887 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255897 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255905 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255914 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255922 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255933 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="53142d59-d966-4f70-9609-f8613c9b2547" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255946 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255953 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255962 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="48312fad-cf56-4728-8f43-ad84bad2537d" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255986 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.255994 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16e3380-5275-4fc3-a2c0-e7b45edf8ca2" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256001 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256008 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256017 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256028 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="757f3a79-c1cb-46c8-8210-bee23a846f45" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256040 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256047 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256059 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:39.256775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.256066 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:39.261693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.261672 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.289342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.278299 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb"] Nov 02 21:45:39.328992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.320390 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:45:39.328992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.320605 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:39.328992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.320677 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:39.328992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.320798 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:45:39.443044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.442004 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339caa72-e44b-46a0-8e4d-500096385890-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.443044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.442165 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtp7\" (UniqueName: \"kubernetes.io/projected/339caa72-e44b-46a0-8e4d-500096385890-kube-api-access-hrtp7\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.443044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.442289 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339caa72-e44b-46a0-8e4d-500096385890-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.543769 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.543735 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtp7\" (UniqueName: \"kubernetes.io/projected/339caa72-e44b-46a0-8e4d-500096385890-kube-api-access-hrtp7\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.543952 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.543836 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339caa72-e44b-46a0-8e4d-500096385890-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.544013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.543956 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339caa72-e44b-46a0-8e4d-500096385890-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.544729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.544701 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339caa72-e44b-46a0-8e4d-500096385890-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.546386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.546363 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339caa72-e44b-46a0-8e4d-500096385890-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.621099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.619631 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtp7\" (UniqueName: \"kubernetes.io/projected/339caa72-e44b-46a0-8e4d-500096385890-kube-api-access-hrtp7\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:39.657021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:39.656992 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:40.108991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:40.108385 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb"] Nov 02 21:45:40.144426 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:40.143634 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339caa72_e44b_46a0_8e4d_500096385890.slice/crio-3d1b0796488ee904e5ef1201ae11c4686ccc89e51bf850de706cb518188f74d0 WatchSource:0}: Error finding container 3d1b0796488ee904e5ef1201ae11c4686ccc89e51bf850de706cb518188f74d0: Status 404 returned error can't find the container with id 3d1b0796488ee904e5ef1201ae11c4686ccc89e51bf850de706cb518188f74d0 Nov 02 21:45:40.616928 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:40.616885 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:45:40.651479 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:40.651444 4441 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:45:40.651635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:40.651491 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:45:40.875206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:40.875166 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" event={"ID":"339caa72-e44b-46a0-8e4d-500096385890","Type":"ContainerStarted","Data":"bcae2283734f3ebb89405917211c3ccd129db97ab84605fa2581d509fd8f8864"} Nov 02 21:45:40.875409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:40.875231 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" event={"ID":"339caa72-e44b-46a0-8e4d-500096385890","Type":"ContainerStarted","Data":"3d1b0796488ee904e5ef1201ae11c4686ccc89e51bf850de706cb518188f74d0"} Nov 02 21:45:40.986990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:40.986426 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" podStartSLOduration=1.986403683 podStartE2EDuration="1.986403683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:40.986047119 +0000 UTC m=+1970.541675456" watchObservedRunningTime="2026-11-02 21:45:40.986403683 +0000 UTC m=+1970.542032025" Nov 02 21:45:41.889954 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:41.889902 4441 generic.go:356] "Generic (PLEG): container finished" podID="339caa72-e44b-46a0-8e4d-500096385890" containerID="bcae2283734f3ebb89405917211c3ccd129db97ab84605fa2581d509fd8f8864" exitCode=0 Nov 02 21:45:41.890449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:41.890007 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" event={"ID":"339caa72-e44b-46a0-8e4d-500096385890","Type":"ContainerDied","Data":"bcae2283734f3ebb89405917211c3ccd129db97ab84605fa2581d509fd8f8864"} Nov 02 21:45:42.232583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:42.232501 4441 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:45:42.232718 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:42.232570 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:45:42.953739 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:42.953701 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bc6189_eae4_4b30_9822_2d890886aa31.slice/crio-a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:45:43.142528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.142162 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:43.221998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.221214 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/339caa72-e44b-46a0-8e4d-500096385890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339caa72-e44b-46a0-8e4d-500096385890-maas-api-xks-ca\") pod \"339caa72-e44b-46a0-8e4d-500096385890\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " Nov 02 21:45:43.221998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.221430 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339caa72-e44b-46a0-8e4d-500096385890-kube-api-access-hrtp7\" (UniqueName: \"kubernetes.io/projected/339caa72-e44b-46a0-8e4d-500096385890-kube-api-access-hrtp7\") pod \"339caa72-e44b-46a0-8e4d-500096385890\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " Nov 02 21:45:43.221998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.221528 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339caa72-e44b-46a0-8e4d-500096385890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339caa72-e44b-46a0-8e4d-500096385890-maas-api-openshift-ca\") pod \"339caa72-e44b-46a0-8e4d-500096385890\" (UID: \"339caa72-e44b-46a0-8e4d-500096385890\") " Nov 02 21:45:43.222844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.222574 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339caa72-e44b-46a0-8e4d-500096385890-maas-api-openshift-ca" pod "339caa72-e44b-46a0-8e4d-500096385890" (UID: "339caa72-e44b-46a0-8e4d-500096385890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:43.222844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.222804 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339caa72-e44b-46a0-8e4d-500096385890-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.230847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.230808 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339caa72-e44b-46a0-8e4d-500096385890-maas-api-xks-ca" pod "339caa72-e44b-46a0-8e4d-500096385890" (UID: "339caa72-e44b-46a0-8e4d-500096385890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:43.232707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.232668 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339caa72-e44b-46a0-8e4d-500096385890-kube-api-access-hrtp7" pod "339caa72-e44b-46a0-8e4d-500096385890" (UID: "339caa72-e44b-46a0-8e4d-500096385890"). InnerVolumeSpecName "kube-api-access-hrtp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:43.262350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.262317 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.323952 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-podinfo\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.324470 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-socket\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.324633 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-certs\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.324751 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-envoy\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.324840 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-socket" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.324913 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-data\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325098 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-certs" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325057 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-token\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-token\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325179 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-credential-socket\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325268 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7bc6189-eae4-4b30-9822-2d890886aa31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bc6189-eae4-4b30-9822-2d890886aa31-istiod-ca-cert\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325298 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-data" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325331 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-kube-api-access-9qkfr\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-kube-api-access-9qkfr\") pod \"e7bc6189-eae4-4b30-9822-2d890886aa31\" (UID: \"e7bc6189-eae4-4b30-9822-2d890886aa31\") " Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325472 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-credential-socket" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:43.325865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.325698 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bc6189-eae4-4b30-9822-2d890886aa31-istiod-ca-cert" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.326282 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.326306 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.326320 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.326335 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339caa72-e44b-46a0-8e4d-500096385890-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.326350 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.326364 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bc6189-eae4-4b30-9822-2d890886aa31-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.326378 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrtp7\" (UniqueName: \"kubernetes.io/projected/339caa72-e44b-46a0-8e4d-500096385890-kube-api-access-hrtp7\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.327119 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-podinfo" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.328662 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-token" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:43.331408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.331342 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-envoy" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:43.332090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.332056 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-kube-api-access-9qkfr" pod "e7bc6189-eae4-4b30-9822-2d890886aa31" (UID: "e7bc6189-eae4-4b30-9822-2d890886aa31"). InnerVolumeSpecName "kube-api-access-9qkfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:43.428195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.428154 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.428195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.428196 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkfr\" (UniqueName: \"kubernetes.io/projected/e7bc6189-eae4-4b30-9822-2d890886aa31-kube-api-access-9qkfr\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.428468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.428210 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.428468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.428225 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bc6189-eae4-4b30-9822-2d890886aa31-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:43.914018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.913981 4441 generic.go:356] "Generic (PLEG): container finished" podID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerID="a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69" exitCode=0 Nov 02 21:45:43.914156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.914097 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" Nov 02 21:45:43.914156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.914106 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" event={"ID":"e7bc6189-eae4-4b30-9822-2d890886aa31","Type":"ContainerDied","Data":"a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69"} Nov 02 21:45:43.914279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.914165 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd" event={"ID":"e7bc6189-eae4-4b30-9822-2d890886aa31","Type":"ContainerDied","Data":"85a1033febb41c3b4ac2f390211b8a323e8623c95184ed8a6e56040263ef184c"} Nov 02 21:45:43.914279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.914191 4441 scope.go:122] "RemoveContainer" containerID="a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69" Nov 02 21:45:43.916208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.916187 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" Nov 02 21:45:43.916303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.916215 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb" event={"ID":"339caa72-e44b-46a0-8e4d-500096385890","Type":"ContainerDied","Data":"3d1b0796488ee904e5ef1201ae11c4686ccc89e51bf850de706cb518188f74d0"} Nov 02 21:45:43.916303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:43.916258 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d1b0796488ee904e5ef1201ae11c4686ccc89e51bf850de706cb518188f74d0" Nov 02 21:45:44.076311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.076278 4441 scope.go:122] "RemoveContainer" containerID="a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69" Nov 02 21:45:44.080522 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:44.080490 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69\": container with ID starting with a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69 not found: ID does not exist" containerID="a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69" Nov 02 21:45:44.080750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.080718 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69"} err="failed to get container status \"a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69\": rpc error: code = NotFound desc = could not find container \"a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69\": container with ID starting with a6d00dd3d35c9e4b1c1b5777f447efb12a42305ec55cb8d0b0dbe6dbff01de69 not found: ID does not exist" Nov 02 21:45:44.223055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.215539 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd"] Nov 02 21:45:44.223467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.223431 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh"] Nov 02 21:45:44.224486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224459 4441 cpu_manager.go:414] "containerMap: removing container" podUID="339caa72-e44b-46a0-8e4d-500096385890" containerName="revoke-keys" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224487 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339caa72-e44b-46a0-8e4d-500096385890" containerName="revoke-keys" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224507 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224516 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224533 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224541 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224558 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224564 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e37857-599f-4d61-9908-0d0b57269498" containerName="cleanup" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224594 4441 cpu_manager.go:414] "containerMap: removing container" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:44.224604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224603 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224614 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224621 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224669 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224677 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224703 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224710 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5862b7-d0b9-44df-ab25-5c562cae664a" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224852 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" containerName="istio-proxy" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224872 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa2c34b-ff3f-40d6-afad-3172880687f7" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224884 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dae9676-856a-44a5-904e-e32d2a114e28" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224893 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc85c55-68e7-40d8-b830-eee11c38de6c" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224904 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc41514-68ad-4d9d-bb6c-9ccf26358703" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224915 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1aae5a-96d6-43b1-83cb-85d3a097faea" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224925 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="339caa72-e44b-46a0-8e4d-500096385890" containerName="revoke-keys" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224934 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="096f83eb-1617-4613-b023-864f9cb54f53" containerName="cleanup" Nov 02 21:45:44.225066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.224943 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c376f194-53a4-4d9f-bf63-dadcdbf79542" containerName="cleanup" Nov 02 21:45:44.231715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.231465 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.264080 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.263247 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-nmbnd"] Nov 02 21:45:44.281336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.281307 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:44.281648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.281631 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:45:44.281875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.281860 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:45:44.282099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.282086 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:44.283208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.283173 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh"] Nov 02 21:45:44.283424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.283393 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6v8p\" (UniqueName: \"kubernetes.io/projected/df719e93-4ca7-4a6b-a317-aeab83d56223-kube-api-access-n6v8p\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.283507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.283446 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.283628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.283609 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.394030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.389363 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.394030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.389596 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6v8p\" (UniqueName: \"kubernetes.io/projected/df719e93-4ca7-4a6b-a317-aeab83d56223-kube-api-access-n6v8p\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.394030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.389632 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.394030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.390575 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.394030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.393366 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.447998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.447887 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:45:44.465350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.450737 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6v8p\" (UniqueName: \"kubernetes.io/projected/df719e93-4ca7-4a6b-a317-aeab83d56223-kube-api-access-n6v8p\") pod \"maas-api-revoke-sub-35031903-ccwvh\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.486635 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.486606 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:45:44.490800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.490775 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-certs\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.490922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.490813 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-kube-api-access-p22gj\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-kube-api-access-p22gj\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.490922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.490833 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37bcec41-697d-40da-be0c-0c31d967d59f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bcec41-697d-40da-be0c-0c31d967d59f-istio-podinfo\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.490922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.490859 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-envoy\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.490935 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-certs\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.491125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.490952 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bcec41-697d-40da-be0c-0c31d967d59f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bcec41-697d-40da-be0c-0c31d967d59f-istiod-ca-cert\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.491125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.490989 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51771762-996a-41a8-b5a1-8dabdd4c8524-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51771762-996a-41a8-b5a1-8dabdd4c8524-istiod-ca-cert\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491058 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-istio-token\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-istio-token\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.491125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491076 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-token\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-token\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491107 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-data\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491148 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-kube-api-access-vw9pj\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-kube-api-access-vw9pj\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491224 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-podinfo\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491260 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-data\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.491356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491287 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-envoy\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.491473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491354 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-credential-socket\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.491473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491428 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-socket\") pod \"37bcec41-697d-40da-be0c-0c31d967d59f\" (UID: \"37bcec41-697d-40da-be0c-0c31d967d59f\") " Nov 02 21:45:44.491536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491489 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-credential-socket\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491517 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-socket\") pod \"51771762-996a-41a8-b5a1-8dabdd4c8524\" (UID: \"51771762-996a-41a8-b5a1-8dabdd4c8524\") " Nov 02 21:45:44.491600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491551 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bcec41-697d-40da-be0c-0c31d967d59f-istiod-ca-cert" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:44.491842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.491813 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-certs" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.492075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492054 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-socket" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.492223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492088 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-certs" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.492328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492304 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-data" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.492394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492329 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.492394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492352 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.492394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492369 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.492394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492384 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bcec41-697d-40da-be0c-0c31d967d59f-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.492643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.492622 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-data" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.493389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.493366 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-istio-token" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:44.493651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.493632 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51771762-996a-41a8-b5a1-8dabdd4c8524-istiod-ca-cert" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:44.493817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.493796 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-socket" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.494063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.494046 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-credential-socket" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.494193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.494154 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-credential-socket" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.494277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.494264 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-envoy" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.494700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.494675 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37bcec41-697d-40da-be0c-0c31d967d59f-istio-podinfo" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:44.494791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.494714 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-envoy" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.494791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.494726 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-kube-api-access-p22gj" pod "37bcec41-697d-40da-be0c-0c31d967d59f" (UID: "37bcec41-697d-40da-be0c-0c31d967d59f"). InnerVolumeSpecName "kube-api-access-p22gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:44.495023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.495000 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-podinfo" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:44.495639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.495611 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-kube-api-access-vw9pj" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "kube-api-access-vw9pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:44.496036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.496012 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-token" pod "51771762-996a-41a8-b5a1-8dabdd4c8524" (UID: "51771762-996a-41a8-b5a1-8dabdd4c8524"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:44.555792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.555751 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:44.593740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593688 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51771762-996a-41a8-b5a1-8dabdd4c8524-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593742 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593759 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593773 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593787 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9pj\" (UniqueName: \"kubernetes.io/projected/51771762-996a-41a8-b5a1-8dabdd4c8524-kube-api-access-vw9pj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593801 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593815 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593829 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593842 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593860 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bcec41-697d-40da-be0c-0c31d967d59f-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593874 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593887 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22gj\" (UniqueName: \"kubernetes.io/projected/37bcec41-697d-40da-be0c-0c31d967d59f-kube-api-access-p22gj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593902 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bcec41-697d-40da-be0c-0c31d967d59f-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.593985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.593918 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51771762-996a-41a8-b5a1-8dabdd4c8524-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.601764 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.601734 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7bc6189-eae4-4b30-9822-2d890886aa31" path="/var/lib/kubelet/pods/e7bc6189-eae4-4b30-9822-2d890886aa31/volumes" Nov 02 21:45:44.926516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.926479 4441 generic.go:356] "Generic (PLEG): container finished" podID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerID="b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e" exitCode=0 Nov 02 21:45:44.926698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.926543 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" Nov 02 21:45:44.926698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.926559 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" event={"ID":"51771762-996a-41a8-b5a1-8dabdd4c8524","Type":"ContainerDied","Data":"b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e"} Nov 02 21:45:44.926698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.926617 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc" event={"ID":"51771762-996a-41a8-b5a1-8dabdd4c8524","Type":"ContainerDied","Data":"0f61db41763a11f5e81d85a956e3f7442c99df4e6f8d71bd35bfd676a7597926"} Nov 02 21:45:44.926698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.926647 4441 scope.go:122] "RemoveContainer" containerID="b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e" Nov 02 21:45:44.930035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.929982 4441 generic.go:356] "Generic (PLEG): container finished" podID="37bcec41-697d-40da-be0c-0c31d967d59f" containerID="ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8" exitCode=0 Nov 02 21:45:44.930035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.930018 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" event={"ID":"37bcec41-697d-40da-be0c-0c31d967d59f","Type":"ContainerDied","Data":"ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8"} Nov 02 21:45:44.930177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.930043 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" Nov 02 21:45:44.930177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.930070 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6" event={"ID":"37bcec41-697d-40da-be0c-0c31d967d59f","Type":"ContainerDied","Data":"f01ce8ef9fe33d9db2202c020e85b0c294edd1b4ba37a863f9dcb8df71e55a97"} Nov 02 21:45:44.938252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.938232 4441 scope.go:122] "RemoveContainer" containerID="b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e" Nov 02 21:45:44.938565 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:44.938540 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e\": container with ID starting with b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e not found: ID does not exist" containerID="b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e" Nov 02 21:45:44.938652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.938578 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e"} err="failed to get container status \"b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e\": rpc error: code = NotFound desc = could not find container \"b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e\": container with ID starting with b55943e3400cb4f904921fb98410f0595cbc7fd983a5b286ceacf725d8ee426e not found: ID does not exist" Nov 02 21:45:44.938652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.938602 4441 scope.go:122] "RemoveContainer" containerID="ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8" Nov 02 21:45:44.950684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.950668 4441 scope.go:122] "RemoveContainer" containerID="ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8" Nov 02 21:45:44.951008 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:44.950946 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8\": container with ID starting with ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8 not found: ID does not exist" containerID="ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8" Nov 02 21:45:44.951104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.951019 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8"} err="failed to get container status \"ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8\": rpc error: code = NotFound desc = could not find container \"ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8\": container with ID starting with ba49e3c3882337931ec21c1e0d3629f41c0cbc7b0feebdc4172c5a8ba32773b8 not found: ID does not exist" Nov 02 21:45:44.979114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:44.979038 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh"] Nov 02 21:45:44.984402 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:44.984374 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf719e93_4ca7_4a6b_a317_aeab83d56223.slice/crio-b614815bd580e15869a6b7359b69b3a16d748d6a6b621b7fbff6c50c2b71b1b5 WatchSource:0}: Error finding container b614815bd580e15869a6b7359b69b3a16d748d6a6b621b7fbff6c50c2b71b1b5: Status 404 returned error can't find the container with id b614815bd580e15869a6b7359b69b3a16d748d6a6b621b7fbff6c50c2b71b1b5 Nov 02 21:45:45.089368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:45.089334 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6"] Nov 02 21:45:45.093930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:45.093898 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-tzgp6"] Nov 02 21:45:45.197550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:45.197506 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc"] Nov 02 21:45:45.202121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:45.202096 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5kdtc"] Nov 02 21:45:45.352850 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:45.352812 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:45.940213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:45.940176 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" event={"ID":"df719e93-4ca7-4a6b-a317-aeab83d56223","Type":"ContainerStarted","Data":"9943101b88c65fae8b39513d40a9961d5767bdda9d93693b36932e60a12fd14d"} Nov 02 21:45:45.940410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:45.940239 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" event={"ID":"df719e93-4ca7-4a6b-a317-aeab83d56223","Type":"ContainerStarted","Data":"b614815bd580e15869a6b7359b69b3a16d748d6a6b621b7fbff6c50c2b71b1b5"} Nov 02 21:45:46.050245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:46.050192 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" podStartSLOduration=2.050178175 podStartE2EDuration="2.050178175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:46.047402005 +0000 UTC m=+1975.603030342" watchObservedRunningTime="2026-11-02 21:45:46.050178175 +0000 UTC m=+1975.605806511" Nov 02 21:45:46.585774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:46.585738 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" path="/var/lib/kubelet/pods/37bcec41-697d-40da-be0c-0c31d967d59f/volumes" Nov 02 21:45:46.586326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:46.586309 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" path="/var/lib/kubelet/pods/51771762-996a-41a8-b5a1-8dabdd4c8524/volumes" Nov 02 21:45:46.952387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:46.952299 4441 generic.go:356] "Generic (PLEG): container finished" podID="df719e93-4ca7-4a6b-a317-aeab83d56223" containerID="9943101b88c65fae8b39513d40a9961d5767bdda9d93693b36932e60a12fd14d" exitCode=0 Nov 02 21:45:46.952387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:46.952339 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" event={"ID":"df719e93-4ca7-4a6b-a317-aeab83d56223","Type":"ContainerDied","Data":"9943101b88c65fae8b39513d40a9961d5767bdda9d93693b36932e60a12fd14d"} Nov 02 21:45:48.140912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.140889 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:48.253155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.253067 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-openshift-ca\") pod \"df719e93-4ca7-4a6b-a317-aeab83d56223\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " Nov 02 21:45:48.253324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.253248 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-xks-ca\") pod \"df719e93-4ca7-4a6b-a317-aeab83d56223\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " Nov 02 21:45:48.253388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.253363 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df719e93-4ca7-4a6b-a317-aeab83d56223-kube-api-access-n6v8p\" (UniqueName: \"kubernetes.io/projected/df719e93-4ca7-4a6b-a317-aeab83d56223-kube-api-access-n6v8p\") pod \"df719e93-4ca7-4a6b-a317-aeab83d56223\" (UID: \"df719e93-4ca7-4a6b-a317-aeab83d56223\") " Nov 02 21:45:48.253823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.253543 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-openshift-ca" pod "df719e93-4ca7-4a6b-a317-aeab83d56223" (UID: "df719e93-4ca7-4a6b-a317-aeab83d56223"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:48.254034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.254013 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:48.255723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.255697 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df719e93-4ca7-4a6b-a317-aeab83d56223-kube-api-access-n6v8p" pod "df719e93-4ca7-4a6b-a317-aeab83d56223" (UID: "df719e93-4ca7-4a6b-a317-aeab83d56223"). InnerVolumeSpecName "kube-api-access-n6v8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:48.255849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.255793 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-xks-ca" pod "df719e93-4ca7-4a6b-a317-aeab83d56223" (UID: "df719e93-4ca7-4a6b-a317-aeab83d56223"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:48.355354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.355317 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df719e93-4ca7-4a6b-a317-aeab83d56223-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:48.355354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.355344 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6v8p\" (UniqueName: \"kubernetes.io/projected/df719e93-4ca7-4a6b-a317-aeab83d56223-kube-api-access-n6v8p\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:48.972565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.972478 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" event={"ID":"df719e93-4ca7-4a6b-a317-aeab83d56223","Type":"ContainerDied","Data":"b614815bd580e15869a6b7359b69b3a16d748d6a6b621b7fbff6c50c2b71b1b5"} Nov 02 21:45:48.972565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.972528 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh" Nov 02 21:45:48.972771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:48.972533 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b614815bd580e15869a6b7359b69b3a16d748d6a6b621b7fbff6c50c2b71b1b5" Nov 02 21:45:49.322995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.322935 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8"] Nov 02 21:45:49.323599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323582 4441 cpu_manager.go:414] "containerMap: removing container" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" Nov 02 21:45:49.323644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323600 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" Nov 02 21:45:49.323644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323637 4441 cpu_manager.go:414] "containerMap: removing container" podUID="df719e93-4ca7-4a6b-a317-aeab83d56223" containerName="revoke-keys" Nov 02 21:45:49.323644 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323643 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df719e93-4ca7-4a6b-a317-aeab83d56223" containerName="revoke-keys" Nov 02 21:45:49.323748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323661 4441 cpu_manager.go:414] "containerMap: removing container" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" Nov 02 21:45:49.323748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323668 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" Nov 02 21:45:49.323812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323763 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="df719e93-4ca7-4a6b-a317-aeab83d56223" containerName="revoke-keys" Nov 02 21:45:49.323812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323772 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="51771762-996a-41a8-b5a1-8dabdd4c8524" containerName="istio-proxy" Nov 02 21:45:49.323812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.323780 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bcec41-697d-40da-be0c-0c31d967d59f" containerName="istio-proxy" Nov 02 21:45:49.327637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.327620 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.334521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.334498 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj"] Nov 02 21:45:49.339065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.339046 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.369964 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.369934 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ssw\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-kube-api-access-c2ssw\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370050 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503d1f35-2825-4893-aacb-3070b61de20b-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370110 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370243 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370281 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370335 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370442 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370479 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.370580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.370567 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503d1f35-2825-4893-aacb-3070b61de20b-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.377231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.377202 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-5451a5-openshift-default-dockercfg-7jfx2\"" Nov 02 21:45:49.421550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.421508 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8"] Nov 02 21:45:49.431024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.430994 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj"] Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472273 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ssw\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-kube-api-access-c2ssw\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472313 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503d1f35-2825-4893-aacb-3070b61de20b-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472339 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472376 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472415 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472439 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472471 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472498 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472528 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472561 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.472622 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473033 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473338 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913f1206-00cc-4569-b38d-20897a9a2cd8-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473411 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473444 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473486 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473572 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473601 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8b7\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-kube-api-access-lv8b7\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473636 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503d1f35-2825-4893-aacb-3070b61de20b-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.474014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.473959 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.475274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.474119 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.475274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.474378 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503d1f35-2825-4893-aacb-3070b61de20b-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.481188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.478137 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.482251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.482223 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.488881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.486571 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503d1f35-2825-4893-aacb-3070b61de20b-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.530620 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.530591 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.530752 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.530656 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ssw\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-kube-api-access-c2ssw\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.575523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575439 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.575523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575495 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913f1206-00cc-4569-b38d-20897a9a2cd8-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.575725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575547 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.575725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575644 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.575725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575677 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8b7\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-kube-api-access-lv8b7\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.575887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575790 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.575887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575870 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.576036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.575946 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.576036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.576011 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.576664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.576524 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.576664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.576586 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.576837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.576821 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.577392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.577368 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913f1206-00cc-4569-b38d-20897a9a2cd8-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.578169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.578143 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.579622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.579600 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.579727 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.579653 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.630439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.630408 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8b7\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-kube-api-access-lv8b7\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.631131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.631113 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:49.642425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.642400 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:49.654959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:49.654934 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:50.091736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.091704 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8"] Nov 02 21:45:50.097818 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.096614 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj"] Nov 02 21:45:50.097818 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:50.096706 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913f1206_00cc_4569_b38d_20897a9a2cd8.slice/crio-1635a7e2b8fe8de54223f9048fa3c05a002316ac1bc5d7376cc064e3e457293f WatchSource:0}: Error finding container 1635a7e2b8fe8de54223f9048fa3c05a002316ac1bc5d7376cc064e3e457293f: Status 404 returned error can't find the container with id 1635a7e2b8fe8de54223f9048fa3c05a002316ac1bc5d7376cc064e3e457293f Nov 02 21:45:50.097818 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:45:50.097196 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503d1f35_2825_4893_aacb_3070b61de20b.slice/crio-d4fbcdec61a6fca2e799d1526dc46157a0c969bb45e0e3ca045e5ceee515cfc8 WatchSource:0}: Error finding container d4fbcdec61a6fca2e799d1526dc46157a0c969bb45e0e3ca045e5ceee515cfc8: Status 404 returned error can't find the container with id d4fbcdec61a6fca2e799d1526dc46157a0c969bb45e0e3ca045e5ceee515cfc8 Nov 02 21:45:50.099750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.099718 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:45:50.099864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.099754 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:45:50.099864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.099793 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:45:50.099864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.099812 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:45:50.099864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.099840 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:45:50.100118 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.099840 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:45:50.521215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:50.521139 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.011059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.011018 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" event={"ID":"913f1206-00cc-4569-b38d-20897a9a2cd8","Type":"ContainerStarted","Data":"44c8ba81ed8c48da7c5d5c41c0e546740516da4e62ea2c8d9c8c66b57e4406ab"} Nov 02 21:45:51.011256 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.011118 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" event={"ID":"913f1206-00cc-4569-b38d-20897a9a2cd8","Type":"ContainerStarted","Data":"1635a7e2b8fe8de54223f9048fa3c05a002316ac1bc5d7376cc064e3e457293f"} Nov 02 21:45:51.013395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.013364 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" event={"ID":"503d1f35-2825-4893-aacb-3070b61de20b","Type":"ContainerStarted","Data":"15b03cbef65bd4774e36c29ee2f515fd082dc277d2cfaf576c1358f6076b569d"} Nov 02 21:45:51.013507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.013412 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" event={"ID":"503d1f35-2825-4893-aacb-3070b61de20b","Type":"ContainerStarted","Data":"d4fbcdec61a6fca2e799d1526dc46157a0c969bb45e0e3ca045e5ceee515cfc8"} Nov 02 21:45:51.221845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.221775 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podStartSLOduration=2.221752983 podStartE2EDuration="2.221752983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:51.183349265 +0000 UTC m=+1980.738977626" watchObservedRunningTime="2026-11-02 21:45:51.221752983 +0000 UTC m=+1980.777381323" Nov 02 21:45:51.310139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.308903 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.313995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.313071 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podStartSLOduration=2.313051362 podStartE2EDuration="2.313051362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:51.296067563 +0000 UTC m=+1980.851695901" watchObservedRunningTime="2026-11-02 21:45:51.313051362 +0000 UTC m=+1980.868679701" Nov 02 21:45:51.477991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.459733 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.626044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.616824 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.642937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.642906 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:51.648662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.648625 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 21:45:51.648831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.648692 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 21:45:51.655198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.655178 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:51.657083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.657056 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 21:45:51.657194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:51.657094 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 21:45:52.159851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.159817 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql"] Nov 02 21:45:52.160099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.160057 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" containerID="cri-o://5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d" gracePeriod=30 Nov 02 21:45:52.248245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.248207 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld"] Nov 02 21:45:52.249094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.248433 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" containerID="cri-o://1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442" gracePeriod=30 Nov 02 21:45:52.491051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.489651 4441 scope.go:122] "RemoveContainer" containerID="707b09bab7b293ef1f49d35858fb1aaa48c0f22f73c9ffa2465242679bd0cccb" Nov 02 21:45:52.654472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.654433 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 21:45:52.655092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.655053 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 21:45:52.663059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.657913 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 21:45:52.663267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:52.663242 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 21:45:53.643579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:53.643491 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 21:45:53.643579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:53.643548 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 21:45:53.656164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:53.656121 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 21:45:53.656507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:53.656185 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 21:45:54.643468 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:54.643427 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 21:45:54.643653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:54.643481 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 21:45:54.655537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:54.655510 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 21:45:54.655675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:54.655554 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 21:45:55.642810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:55.642764 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 21:45:55.643291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:55.642846 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 21:45:55.655581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:55.655554 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 21:45:55.655714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:55.655635 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 21:45:55.991017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:55.990919 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:56.334530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.334409 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:45:56.334530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.334458 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:45:56.395594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.395558 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:45:56.395897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.395857 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:45:56.643253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.643108 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 21:45:56.643253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.643179 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 21:45:56.656208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.656118 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 21:45:56.656208 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:56.656169 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 21:45:57.572136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.572107 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:45:57.648495 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.648471 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:57.648850 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.648763 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:57.649482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.649461 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:45:57.655462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.655441 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 21:45:57.655575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.655484 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 21:45:57.664508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.664484 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:45:57.689251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689213 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-token\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-token\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.689452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689310 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-envoy\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.689452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689423 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-credential-socket\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.689577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689489 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-data\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.689577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689530 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-kube-api-access-pprdw\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-kube-api-access-pprdw\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.689577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689561 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-socket\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.689751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689588 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-podinfo\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.690025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.689996 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-data" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.690629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.690585 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-credential-socket" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.691033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.691000 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-socket" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.691257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.691239 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-certs\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.692736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.692716 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istiod-ca-cert\") pod \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\" (UID: \"8b0ba330-cf40-4a1e-b2bb-6a18960f7260\") " Nov 02 21:45:57.693601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.691588 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-certs" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.693601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.691950 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-token" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:57.693601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.692494 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-envoy" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.693601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.693182 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istiod-ca-cert" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:57.693601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.693526 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-podinfo" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:57.694051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.694010 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-kube-api-access-pprdw" pod "8b0ba330-cf40-4a1e-b2bb-6a18960f7260" (UID: "8b0ba330-cf40-4a1e-b2bb-6a18960f7260"). InnerVolumeSpecName "kube-api-access-pprdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:57.694771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.694749 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.695774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.695757 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.695898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.695885 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.696015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.695999 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pprdw\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-kube-api-access-pprdw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.696097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.696087 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.696175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.696164 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.696259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.696248 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.696340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.696328 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.696423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.696412 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0ba330-cf40-4a1e-b2bb-6a18960f7260-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.797008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.796902 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-podinfo\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797018 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-envoy\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797182 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797131 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-socket\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797211 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-kube-api-access-4mltn\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-kube-api-access-4mltn\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797284 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-data\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797322 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-certs\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797355 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istiod-ca-cert\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797461 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-token\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-token\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.797570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.797508 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-credential-socket\") pod \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\" (UID: \"4caabab8-b0ee-4748-a24f-6bdeb1149f93\") " Nov 02 21:45:57.798279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.798185 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-socket" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.798682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.798296 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.798682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.798450 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-data" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.798682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.798591 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-credential-socket" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.798948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.798717 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istiod-ca-cert" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:57.799055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.798957 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-certs" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.800728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.800701 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-kube-api-access-4mltn" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "kube-api-access-4mltn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:57.801369 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.801352 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-token" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:57.802276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.802249 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-envoy" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:57.802353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.802300 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-podinfo" pod "4caabab8-b0ee-4748-a24f-6bdeb1149f93" (UID: "4caabab8-b0ee-4748-a24f-6bdeb1149f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:57.900038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.899997 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mltn\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-kube-api-access-4mltn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.900038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.900042 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.900316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.900059 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.900316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.900073 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.900316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.900086 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.900316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.900100 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.900316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.900115 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.900316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:57.900129 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caabab8-b0ee-4748-a24f-6bdeb1149f93-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:58.261782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.261747 4441 generic.go:356] "Generic (PLEG): container finished" podID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerID="5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d" exitCode=0 Nov 02 21:45:58.262024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.261798 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" Nov 02 21:45:58.262024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.261829 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" event={"ID":"8b0ba330-cf40-4a1e-b2bb-6a18960f7260","Type":"ContainerDied","Data":"5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d"} Nov 02 21:45:58.262024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.261896 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql" event={"ID":"8b0ba330-cf40-4a1e-b2bb-6a18960f7260","Type":"ContainerDied","Data":"ee4d885eff47c51781c9b9affbe377a434cfbb8779bc99cb34a5b21171bc8c76"} Nov 02 21:45:58.262024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.261927 4441 scope.go:122] "RemoveContainer" containerID="5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d" Nov 02 21:45:58.263639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.263611 4441 generic.go:356] "Generic (PLEG): container finished" podID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerID="1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442" exitCode=0 Nov 02 21:45:58.263762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.263638 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" event={"ID":"4caabab8-b0ee-4748-a24f-6bdeb1149f93","Type":"ContainerDied","Data":"1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442"} Nov 02 21:45:58.263762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.263687 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" Nov 02 21:45:58.263762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.263691 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld" event={"ID":"4caabab8-b0ee-4748-a24f-6bdeb1149f93","Type":"ContainerDied","Data":"876c0338ac7722a559d45a329b8c08660eefc0b1d1532dba4608e6de3a88e900"} Nov 02 21:45:58.317142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.317021 4441 scope.go:122] "RemoveContainer" containerID="5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d" Nov 02 21:45:58.317398 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:58.317375 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d\": container with ID starting with 5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d not found: ID does not exist" containerID="5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d" Nov 02 21:45:58.317492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.317409 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d"} err="failed to get container status \"5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d\": rpc error: code = NotFound desc = could not find container \"5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d\": container with ID starting with 5bed0cb40f15f3264c0326f2b4b834bd8ee4324a8f003e8d9f2c1d4f1276042d not found: ID does not exist" Nov 02 21:45:58.317492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.317433 4441 scope.go:122] "RemoveContainer" containerID="1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442" Nov 02 21:45:58.327916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.327897 4441 scope.go:122] "RemoveContainer" containerID="1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442" Nov 02 21:45:58.328206 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:45:58.328185 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442\": container with ID starting with 1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442 not found: ID does not exist" containerID="1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442" Nov 02 21:45:58.328286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.328214 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442"} err="failed to get container status \"1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442\": rpc error: code = NotFound desc = could not find container \"1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442\": container with ID starting with 1da0a952517b62e8df2927b5e4356b236c8101240e61234788a99addbee79442 not found: ID does not exist" Nov 02 21:45:58.424200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.424172 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql"] Nov 02 21:45:58.428470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.428446 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-79tql"] Nov 02 21:45:58.528562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.528471 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld"] Nov 02 21:45:58.532873 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.532846 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-7hdld"] Nov 02 21:45:58.586874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.586837 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" path="/var/lib/kubelet/pods/4caabab8-b0ee-4748-a24f-6bdeb1149f93/volumes" Nov 02 21:45:58.587416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.587400 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" path="/var/lib/kubelet/pods/8b0ba330-cf40-4a1e-b2bb-6a18960f7260/volumes" Nov 02 21:45:58.660305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.660280 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:58.660769 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.660752 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:45:58.661504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:45:58.661474 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:46:01.054264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.054232 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq"] Nov 02 21:46:01.054954 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.054933 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" Nov 02 21:46:01.054954 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.054952 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" Nov 02 21:46:01.055146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.054999 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" Nov 02 21:46:01.055146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.055007 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" Nov 02 21:46:01.055212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.055151 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0ba330-cf40-4a1e-b2bb-6a18960f7260" containerName="istio-proxy" Nov 02 21:46:01.055212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.055168 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4caabab8-b0ee-4748-a24f-6bdeb1149f93" containerName="istio-proxy" Nov 02 21:46:01.060721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.060701 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.099401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.099368 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:01.112401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.112370 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq"] Nov 02 21:46:01.152842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.152510 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:46:01.152842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.152634 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:46:01.152842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.152668 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:46:01.152842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.152747 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:46:01.245729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.245695 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79260734-7804-4c16-a20d-f7a67d726210-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.245905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.245760 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9dn\" (UniqueName: \"kubernetes.io/projected/79260734-7804-4c16-a20d-f7a67d726210-kube-api-access-kv9dn\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.246020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.245937 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79260734-7804-4c16-a20d-f7a67d726210-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.347367 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.347327 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79260734-7804-4c16-a20d-f7a67d726210-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.347571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.347430 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9dn\" (UniqueName: \"kubernetes.io/projected/79260734-7804-4c16-a20d-f7a67d726210-kube-api-access-kv9dn\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.347571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.347533 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79260734-7804-4c16-a20d-f7a67d726210-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.348211 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.348183 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79260734-7804-4c16-a20d-f7a67d726210-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.350087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.350063 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79260734-7804-4c16-a20d-f7a67d726210-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.364310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.364284 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928"] Nov 02 21:46:01.369102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.369080 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.372733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.372708 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2"] Nov 02 21:46:01.377755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.377724 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.405613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.405574 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9dn\" (UniqueName: \"kubernetes.io/projected/79260734-7804-4c16-a20d-f7a67d726210-kube-api-access-kv9dn\") pod \"maas-api-revoke-keys-e2e-api-a-b7107c-w29zq\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.442088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.442051 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928"] Nov 02 21:46:01.474026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.470901 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2"] Nov 02 21:46:01.553712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.553677 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.553863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.553715 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.553863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.553742 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.553863 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.554018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.553931 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.553999 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554036 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554055 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554091 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.554164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554124 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.554164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554143 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8n8\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-kube-api-access-kz8n8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554217 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.554413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554262 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.554413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554299 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.554413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554396 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554449 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554535 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.554610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.554594 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lnjf\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-kube-api-access-4lnjf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.655661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655560 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.655661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655614 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.655661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655638 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655672 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655743 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655786 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655832 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655860 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655887 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.655956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.655926 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656004 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8n8\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-kube-api-access-kz8n8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656055 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656081 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656113 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656149 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656159 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656225 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656232 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656306 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656310 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656381 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656417 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656440 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lnjf\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-kube-api-access-4lnjf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656487 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656504 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.656782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656507 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.656782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656558 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.657114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.656899 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.658662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.658641 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.658924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.658902 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.659056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.658998 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.659276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.659257 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.676716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.676692 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:01.712795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.712766 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.713135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.713111 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lnjf\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-kube-api-access-4lnjf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6swr2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:01.713310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.713235 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8n8\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-kube-api-access-kz8n8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.713619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.713590 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cd928\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.987233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.987156 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:01.995765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:01.995735 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:02.060655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.060628 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq"] Nov 02 21:46:02.304388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.304294 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" event={"ID":"79260734-7804-4c16-a20d-f7a67d726210","Type":"ContainerStarted","Data":"f24de81f0785763bd2c1a9580ce0d82b27f919a93f81fe7d994cfbd86d5a0ef4"} Nov 02 21:46:02.376772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.376677 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928"] Nov 02 21:46:02.384526 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:46:02.384498 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4b83ca_ee48_4d75_aaa4_d58a5f23eb09.slice/crio-42e706865a3ea1c246153fdafe374f449e672c265808df40a6c24cea8f96bc5d WatchSource:0}: Error finding container 42e706865a3ea1c246153fdafe374f449e672c265808df40a6c24cea8f96bc5d: Status 404 returned error can't find the container with id 42e706865a3ea1c246153fdafe374f449e672c265808df40a6c24cea8f96bc5d Nov 02 21:46:02.388097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.388062 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:02.388209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.388148 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:02.388209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.388203 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:02.423134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.423105 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2"] Nov 02 21:46:02.427307 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:46:02.427279 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce53479_cad7_4031_b32f_3b1ba1d5eed2.slice/crio-c17d4078cdb74c439a19922f3885d0c89ae2f9598bfc615d8548a3355c355936 WatchSource:0}: Error finding container c17d4078cdb74c439a19922f3885d0c89ae2f9598bfc615d8548a3355c355936: Status 404 returned error can't find the container with id c17d4078cdb74c439a19922f3885d0c89ae2f9598bfc615d8548a3355c355936 Nov 02 21:46:02.430449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.430415 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:02.430549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.430494 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:02.430549 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:02.430540 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:03.335991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.335880 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" event={"ID":"79260734-7804-4c16-a20d-f7a67d726210","Type":"ContainerStarted","Data":"fc41c32d45bb50a3352af984a0f712e04ab5ef5848602bb46a553ed86100cdcc"} Nov 02 21:46:03.339025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.338946 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" event={"ID":"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09","Type":"ContainerStarted","Data":"fcb0121ff601e6db2db900403079d5b5d8e9d1aaafbe1af345370e14b84212e6"} Nov 02 21:46:03.339025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.339014 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" event={"ID":"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09","Type":"ContainerStarted","Data":"42e706865a3ea1c246153fdafe374f449e672c265808df40a6c24cea8f96bc5d"} Nov 02 21:46:03.341490 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.341460 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" event={"ID":"7ce53479-cad7-4031-b32f-3b1ba1d5eed2","Type":"ContainerStarted","Data":"5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e"} Nov 02 21:46:03.341576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.341502 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" event={"ID":"7ce53479-cad7-4031-b32f-3b1ba1d5eed2","Type":"ContainerStarted","Data":"c17d4078cdb74c439a19922f3885d0c89ae2f9598bfc615d8548a3355c355936"} Nov 02 21:46:03.455276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.455214 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" podStartSLOduration=2.455200046 podStartE2EDuration="2.455200046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:03.45053819 +0000 UTC m=+1993.006166528" watchObservedRunningTime="2026-11-02 21:46:03.455200046 +0000 UTC m=+1993.010828421" Nov 02 21:46:03.601043 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.600796 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podStartSLOduration=2.600773121 podStartE2EDuration="2.600773121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:03.578634255 +0000 UTC m=+1993.134262618" watchObservedRunningTime="2026-11-02 21:46:03.600773121 +0000 UTC m=+1993.156401459" Nov 02 21:46:03.717991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:03.716160 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podStartSLOduration=2.716137118 podStartE2EDuration="2.716137118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:03.694475626 +0000 UTC m=+1993.250103996" watchObservedRunningTime="2026-11-02 21:46:03.716137118 +0000 UTC m=+1993.271765456" Nov 02 21:46:04.002150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.002055 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:04.002525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.002487 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6swr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 02 21:46:04.002704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.002675 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 02 21:46:04.005135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.005032 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:04.012990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.009100 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 21:46:04.012990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.009160 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 21:46:04.353586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.353556 4441 generic.go:356] "Generic (PLEG): container finished" podID="79260734-7804-4c16-a20d-f7a67d726210" containerID="fc41c32d45bb50a3352af984a0f712e04ab5ef5848602bb46a553ed86100cdcc" exitCode=0 Nov 02 21:46:04.354681 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.354658 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" event={"ID":"79260734-7804-4c16-a20d-f7a67d726210","Type":"ContainerDied","Data":"fc41c32d45bb50a3352af984a0f712e04ab5ef5848602bb46a553ed86100cdcc"} Nov 02 21:46:04.727296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.727129 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.855748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.855710 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.996046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.995340 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 21:46:04.996046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.995420 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 21:46:04.996297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.996074 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6swr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 02 21:46:04.996297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:04.996113 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 02 21:46:05.850065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.850042 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:05.928063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.920491 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79260734-7804-4c16-a20d-f7a67d726210-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79260734-7804-4c16-a20d-f7a67d726210-maas-api-openshift-ca\") pod \"79260734-7804-4c16-a20d-f7a67d726210\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " Nov 02 21:46:05.928063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.920703 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79260734-7804-4c16-a20d-f7a67d726210-kube-api-access-kv9dn\" (UniqueName: \"kubernetes.io/projected/79260734-7804-4c16-a20d-f7a67d726210-kube-api-access-kv9dn\") pod \"79260734-7804-4c16-a20d-f7a67d726210\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " Nov 02 21:46:05.928063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.920857 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79260734-7804-4c16-a20d-f7a67d726210-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79260734-7804-4c16-a20d-f7a67d726210-maas-api-xks-ca\") pod \"79260734-7804-4c16-a20d-f7a67d726210\" (UID: \"79260734-7804-4c16-a20d-f7a67d726210\") " Nov 02 21:46:05.928063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.925438 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79260734-7804-4c16-a20d-f7a67d726210-maas-api-xks-ca" pod "79260734-7804-4c16-a20d-f7a67d726210" (UID: "79260734-7804-4c16-a20d-f7a67d726210"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:05.928063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.925788 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79260734-7804-4c16-a20d-f7a67d726210-maas-api-openshift-ca" pod "79260734-7804-4c16-a20d-f7a67d726210" (UID: "79260734-7804-4c16-a20d-f7a67d726210"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:05.929059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.929026 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79260734-7804-4c16-a20d-f7a67d726210-kube-api-access-kv9dn" pod "79260734-7804-4c16-a20d-f7a67d726210" (UID: "79260734-7804-4c16-a20d-f7a67d726210"). InnerVolumeSpecName "kube-api-access-kv9dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:05.991027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.990989 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 21:46:05.991218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:05.991051 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 21:46:06.015992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.013305 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6swr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 02 21:46:06.015992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.013363 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 02 21:46:06.027079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.024202 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv9dn\" (UniqueName: \"kubernetes.io/projected/79260734-7804-4c16-a20d-f7a67d726210-kube-api-access-kv9dn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:06.027079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.024252 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79260734-7804-4c16-a20d-f7a67d726210-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:06.027079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.024270 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79260734-7804-4c16-a20d-f7a67d726210-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:06.224366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.224153 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:06.432088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.432052 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" Nov 02 21:46:06.432261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.432091 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq" event={"ID":"79260734-7804-4c16-a20d-f7a67d726210","Type":"ContainerDied","Data":"f24de81f0785763bd2c1a9580ce0d82b27f919a93f81fe7d994cfbd86d5a0ef4"} Nov 02 21:46:06.432261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:06.432183 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24de81f0785763bd2c1a9580ce0d82b27f919a93f81fe7d994cfbd86d5a0ef4" Nov 02 21:46:07.007034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.003358 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6swr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 02 21:46:07.007034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.003431 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 02 21:46:07.023141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.023106 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 21:46:07.023306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.023182 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 21:46:07.235598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.235562 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm"] Nov 02 21:46:07.236268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.236253 4441 cpu_manager.go:414] "containerMap: removing container" podUID="79260734-7804-4c16-a20d-f7a67d726210" containerName="revoke-keys" Nov 02 21:46:07.236315 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.236269 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79260734-7804-4c16-a20d-f7a67d726210" containerName="revoke-keys" Nov 02 21:46:07.236386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.236377 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="79260734-7804-4c16-a20d-f7a67d726210" containerName="revoke-keys" Nov 02 21:46:07.239623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.239607 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.293039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.292620 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:46:07.320992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.314453 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm"] Nov 02 21:46:07.333257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.333215 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:46:07.333417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.333298 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:46:07.333417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.333350 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:46:07.362326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.362286 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs24j\" (UniqueName: \"kubernetes.io/projected/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-kube-api-access-rs24j\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.362326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.362324 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.362568 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.362478 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.464249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.464219 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.464488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.464443 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs24j\" (UniqueName: \"kubernetes.io/projected/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-kube-api-access-rs24j\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.464488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.464483 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.465313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.465283 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.467131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.467108 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.533950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.533905 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs24j\" (UniqueName: \"kubernetes.io/projected/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-kube-api-access-rs24j\") pod \"maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:07.557520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:07.557484 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:08.005047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.004369 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 21:46:08.005047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.004430 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 21:46:08.022399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.022364 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6swr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 02 21:46:08.022927 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.022898 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 02 21:46:08.059748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.059715 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm"] Nov 02 21:46:08.063687 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:46:08.063661 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a93ff5_f833_4558_9ae8_5befc8af8cc3.slice/crio-e0aafe50e0f691ba4b244a7b4b7699e786f5bbfd8ed206d55b9cfce76d65022d WatchSource:0}: Error finding container e0aafe50e0f691ba4b244a7b4b7699e786f5bbfd8ed206d55b9cfce76d65022d: Status 404 returned error can't find the container with id e0aafe50e0f691ba4b244a7b4b7699e786f5bbfd8ed206d55b9cfce76d65022d Nov 02 21:46:08.436442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.436410 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d"] Nov 02 21:46:08.436618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.436598 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" containerID="cri-o://398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75" gracePeriod=30 Nov 02 21:46:08.451135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.450746 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" event={"ID":"a9a93ff5-f833-4558-9ae8-5befc8af8cc3","Type":"ContainerStarted","Data":"e0aafe50e0f691ba4b244a7b4b7699e786f5bbfd8ed206d55b9cfce76d65022d"} Nov 02 21:46:08.988257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.988223 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 21:46:08.988416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.988281 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 21:46:08.996223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.996191 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6swr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 02 21:46:08.996353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:08.996236 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 02 21:46:09.463097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:09.463044 4441 generic.go:356] "Generic (PLEG): container finished" podID="a9a93ff5-f833-4558-9ae8-5befc8af8cc3" containerID="e850903cfc42532153587a5d13b325798a749d3044aab04b795a6e09e241e8ce" exitCode=0 Nov 02 21:46:09.463599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:09.463111 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" event={"ID":"a9a93ff5-f833-4558-9ae8-5befc8af8cc3","Type":"ContainerDied","Data":"e850903cfc42532153587a5d13b325798a749d3044aab04b795a6e09e241e8ce"} Nov 02 21:46:09.987726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:09.987682 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 21:46:09.987874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:09.987759 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 21:46:09.996042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:09.996012 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6swr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 02 21:46:09.996131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:09.996062 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 02 21:46:10.529556 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.529519 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:46:10.530034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.529577 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:46:10.653524 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:46:10.653489 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:46:10.669361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.669339 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:10.792039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.789652 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-kube-api-access-rs24j\" (UniqueName: \"kubernetes.io/projected/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-kube-api-access-rs24j\") pod \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " Nov 02 21:46:10.792039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.789723 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-xks-ca\") pod \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " Nov 02 21:46:10.792039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.789840 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-openshift-ca\") pod \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\" (UID: \"a9a93ff5-f833-4558-9ae8-5befc8af8cc3\") " Nov 02 21:46:10.792039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.791501 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-openshift-ca" pod "a9a93ff5-f833-4558-9ae8-5befc8af8cc3" (UID: "a9a93ff5-f833-4558-9ae8-5befc8af8cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:10.833511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.829497 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-xks-ca" pod "a9a93ff5-f833-4558-9ae8-5befc8af8cc3" (UID: "a9a93ff5-f833-4558-9ae8-5befc8af8cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:10.833511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.830466 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-kube-api-access-rs24j" pod "a9a93ff5-f833-4558-9ae8-5befc8af8cc3" (UID: "a9a93ff5-f833-4558-9ae8-5befc8af8cc3"). InnerVolumeSpecName "kube-api-access-rs24j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:10.892614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.892560 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:10.892614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.892608 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs24j\" (UniqueName: \"kubernetes.io/projected/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-kube-api-access-rs24j\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:10.892853 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:10.892625 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a93ff5-f833-4558-9ae8-5befc8af8cc3-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:11.118502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.118466 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:11.514047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.513531 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" event={"ID":"a9a93ff5-f833-4558-9ae8-5befc8af8cc3","Type":"ContainerDied","Data":"e0aafe50e0f691ba4b244a7b4b7699e786f5bbfd8ed206d55b9cfce76d65022d"} Nov 02 21:46:11.514047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.513596 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0aafe50e0f691ba4b244a7b4b7699e786f5bbfd8ed206d55b9cfce76d65022d" Nov 02 21:46:11.514047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.513662 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm" Nov 02 21:46:11.881923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.881878 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:11.882596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.882522 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:11.884031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.883954 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:46:11.902616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.902572 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:11.903767 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.903740 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:11.914231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:11.914205 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:46:12.946421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:12.946388 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:13.070180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.070146 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:13.730112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.730050 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:46:13.750789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.750756 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-kube-api-access-ljrvv\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-kube-api-access-ljrvv\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.750918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.750868 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-envoy\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.750918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.750902 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fbb463-48a0-41fa-877e-1bbaab353269-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fbb463-48a0-41fa-877e-1bbaab353269-istiod-ca-cert\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.751141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.750943 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-certs\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.751207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751148 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-data\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.751207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751183 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-socket\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.751317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751218 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59fbb463-48a0-41fa-877e-1bbaab353269-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fbb463-48a0-41fa-877e-1bbaab353269-istio-podinfo\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.751317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751217 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-certs" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:13.751317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751258 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-istio-token\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-istio-token\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.751317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751294 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-credential-socket\") pod \"59fbb463-48a0-41fa-877e-1bbaab353269\" (UID: \"59fbb463-48a0-41fa-877e-1bbaab353269\") " Nov 02 21:46:13.751317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751296 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fbb463-48a0-41fa-877e-1bbaab353269-istiod-ca-cert" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:13.751572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751377 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-data" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:13.751572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751426 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-socket" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:13.751682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751592 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-credential-socket" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:13.751951 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751930 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fbb463-48a0-41fa-877e-1bbaab353269-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.752046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.751957 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.752046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.752005 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.752046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.752023 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.752046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.752036 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.753281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.753261 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-kube-api-access-ljrvv" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "kube-api-access-ljrvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:13.753366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.753264 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-envoy" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:13.753538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.753521 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-istio-token" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:13.753886 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.753869 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59fbb463-48a0-41fa-877e-1bbaab353269-istio-podinfo" pod "59fbb463-48a0-41fa-877e-1bbaab353269" (UID: "59fbb463-48a0-41fa-877e-1bbaab353269"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:13.853740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.853707 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrvv\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-kube-api-access-ljrvv\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.853740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.853740 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fbb463-48a0-41fa-877e-1bbaab353269-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.853740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.853756 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fbb463-48a0-41fa-877e-1bbaab353269-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.853955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:13.853769 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fbb463-48a0-41fa-877e-1bbaab353269-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:14.540295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.540261 4441 generic.go:356] "Generic (PLEG): container finished" podID="59fbb463-48a0-41fa-877e-1bbaab353269" containerID="398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75" exitCode=0 Nov 02 21:46:14.540721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.540316 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" Nov 02 21:46:14.540721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.540338 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" event={"ID":"59fbb463-48a0-41fa-877e-1bbaab353269","Type":"ContainerDied","Data":"398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75"} Nov 02 21:46:14.540721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.540393 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d" event={"ID":"59fbb463-48a0-41fa-877e-1bbaab353269","Type":"ContainerDied","Data":"3363c857ae21a2bdbd3e4d27fd044f50cb081b346b57619ba747cce8f91c1d46"} Nov 02 21:46:14.540721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.540415 4441 scope.go:122] "RemoveContainer" containerID="398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75" Nov 02 21:46:14.552391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.552369 4441 scope.go:122] "RemoveContainer" containerID="398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75" Nov 02 21:46:14.552669 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:46:14.552650 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75\": container with ID starting with 398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75 not found: ID does not exist" containerID="398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75" Nov 02 21:46:14.552776 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.552674 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75"} err="failed to get container status \"398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75\": rpc error: code = NotFound desc = could not find container \"398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75\": container with ID starting with 398467b3250a8da880480d436c1ab431c6a9a266faac33b0e5bcd110e00f7f75 not found: ID does not exist" Nov 02 21:46:14.745296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.745266 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d"] Nov 02 21:46:14.793002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:14.792917 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-fsc7d"] Nov 02 21:46:16.221265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:16.221228 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:16.586982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:16.586941 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" path="/var/lib/kubelet/pods/59fbb463-48a0-41fa-877e-1bbaab353269/volumes" Nov 02 21:46:18.127862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:18.127826 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:18.252794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:18.252756 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:19.577638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.577602 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx"] Nov 02 21:46:19.578598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.578563 4441 cpu_manager.go:414] "containerMap: removing container" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" Nov 02 21:46:19.578598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.578589 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" Nov 02 21:46:19.578775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.578614 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a9a93ff5-f833-4558-9ae8-5befc8af8cc3" containerName="revoke-keys" Nov 02 21:46:19.578775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.578623 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a93ff5-f833-4558-9ae8-5befc8af8cc3" containerName="revoke-keys" Nov 02 21:46:19.578908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.578890 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fbb463-48a0-41fa-877e-1bbaab353269" containerName="istio-proxy" Nov 02 21:46:19.578950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.578915 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a93ff5-f833-4558-9ae8-5befc8af8cc3" containerName="revoke-keys" Nov 02 21:46:19.592961 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.592932 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.680828 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681402 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-kube-api-access-88vsf\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681459 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681594 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681743 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681722 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681756 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6722f506-00e1-4596-aaab-34e7a8da562b-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681830 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.681917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681884 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.682046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.681929 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6722f506-00e1-4596-aaab-34e7a8da562b-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.714373 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.714342 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx"] Nov 02 21:46:19.783712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.783680 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.783893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.783751 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-kube-api-access-88vsf\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.783893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.783784 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.783893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.783839 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.783893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.783890 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.783913 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6722f506-00e1-4596-aaab-34e7a8da562b-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.783979 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.784019 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.784063 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6722f506-00e1-4596-aaab-34e7a8da562b-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.784226 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.784264 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.784335 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.784817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.784796 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.785022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.784998 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6722f506-00e1-4596-aaab-34e7a8da562b-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.786893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.786868 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6722f506-00e1-4596-aaab-34e7a8da562b-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.787011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.786893 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.837762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.837709 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-kube-api-access-88vsf\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.837762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.837747 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:19.916321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:19.916299 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:20.352114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.352076 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx"] Nov 02 21:46:20.357089 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:46:20.357060 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6722f506_00e1_4596_aaab_34e7a8da562b.slice/crio-2568d850519bcffb2eb74defbf43d6c0844017aaaca4f1c8ab7a03fc85701fe4 WatchSource:0}: Error finding container 2568d850519bcffb2eb74defbf43d6c0844017aaaca4f1c8ab7a03fc85701fe4: Status 404 returned error can't find the container with id 2568d850519bcffb2eb74defbf43d6c0844017aaaca4f1c8ab7a03fc85701fe4 Nov 02 21:46:20.360112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.360078 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:20.360196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.360147 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:20.360196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.360176 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:46:20.622918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.622837 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" event={"ID":"6722f506-00e1-4596-aaab-34e7a8da562b","Type":"ContainerStarted","Data":"18ee7b2c45e544d11ad421925f784af0cff36de74bf7dbde61d18c4be41c28ae"} Nov 02 21:46:20.622918 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.622894 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" event={"ID":"6722f506-00e1-4596-aaab-34e7a8da562b","Type":"ContainerStarted","Data":"2568d850519bcffb2eb74defbf43d6c0844017aaaca4f1c8ab7a03fc85701fe4"} Nov 02 21:46:20.769413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.769346 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" podStartSLOduration=1.769326683 podStartE2EDuration="1.769326683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:20.732796209 +0000 UTC m=+2010.288424537" watchObservedRunningTime="2026-11-02 21:46:20.769326683 +0000 UTC m=+2010.324955023" Nov 02 21:46:20.916845 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:20.916768 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:21.253303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:21.253214 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:21.919523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:21.919489 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 02 21:46:21.919901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:21.919550 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 02 21:46:22.916643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:22.916610 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 02 21:46:22.916834 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:22.916665 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 02 21:46:23.802238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:23.802201 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7"] Nov 02 21:46:23.802742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:23.802485 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" containerID="cri-o://014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb" gracePeriod=30 Nov 02 21:46:23.845044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:23.845013 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz"] Nov 02 21:46:23.845327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:23.845211 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" containerID="cri-o://d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1" gracePeriod=30 Nov 02 21:46:23.917241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:23.917187 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 02 21:46:23.917428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:23.917249 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 02 21:46:24.917196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:24.917158 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 02 21:46:24.917598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:24.917221 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 02 21:46:25.916537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:25.916507 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 02 21:46:25.916775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:25.916754 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 02 21:46:26.597495 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:26.597465 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:26.922042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:26.921947 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:26.922439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:26.922418 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:26.923156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:26.923139 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:46:27.057675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:27.057638 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:46:27.057855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:27.057695 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:46:27.479042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:27.479006 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:46:27.479234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:27.479082 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:46:29.160668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.160632 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:46:29.216629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.216597 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:46:29.255648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.255219 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-token\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-token\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.255825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.255702 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-certs\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.256131 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256105 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-certs" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.256228 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256143 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-data\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.256289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256251 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-envoy\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.256346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256298 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-podinfo\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.256400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256388 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-kube-api-access-cnj7p\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-kube-api-access-cnj7p\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.258217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256480 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-socket\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.258217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256488 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-data" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.258217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256607 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb843693-22ff-4e1b-8048-cb2b971339b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb843693-22ff-4e1b-8048-cb2b971339b4-istiod-ca-cert\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.258217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.256697 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-credential-socket\") pod \"cb843693-22ff-4e1b-8048-cb2b971339b4\" (UID: \"cb843693-22ff-4e1b-8048-cb2b971339b4\") " Nov 02 21:46:29.259846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.259563 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-podinfo" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:29.259846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.259596 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-token" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:29.259846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.259614 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.259846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.259637 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.260206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.260014 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-socket" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.260206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.260093 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-credential-socket" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.260393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.260372 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb843693-22ff-4e1b-8048-cb2b971339b4-istiod-ca-cert" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:29.267307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.267260 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-envoy" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.271021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.268789 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-kube-api-access-cnj7p" pod "cb843693-22ff-4e1b-8048-cb2b971339b4" (UID: "cb843693-22ff-4e1b-8048-cb2b971339b4"). InnerVolumeSpecName "kube-api-access-cnj7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:29.360542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.360507 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-credential-socket\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.360723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.360643 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-kube-api-access-mg5p8\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-kube-api-access-mg5p8\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.360802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.360719 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istiod-ca-cert\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.360864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.360832 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-credential-socket" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.360925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.360889 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-envoy\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.361017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.360996 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-podinfo\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.361191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.361053 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-data\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.361191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.361118 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istiod-ca-cert" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:29.361191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.361180 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-socket\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.361380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.361287 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-certs\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.361419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.361393 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-token\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-token\") pod \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\" (UID: \"d45ffb6e-c4bc-495a-b50d-42c41d6842fe\") " Nov 02 21:46:29.361831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.361680 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-socket" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362001 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-certs" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362101 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362123 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362139 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnj7p\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-kube-api-access-cnj7p\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362153 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362166 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362180 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb843693-22ff-4e1b-8048-cb2b971339b4-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362193 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362207 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb843693-22ff-4e1b-8048-cb2b971339b4-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362221 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb843693-22ff-4e1b-8048-cb2b971339b4-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.362240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.362234 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.363090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.363069 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-data" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.364112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.364087 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-podinfo" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:29.364217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.364177 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-kube-api-access-mg5p8" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "kube-api-access-mg5p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:29.364772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.364743 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-envoy" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:29.365502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.365464 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-token" pod "d45ffb6e-c4bc-495a-b50d-42c41d6842fe" (UID: "d45ffb6e-c4bc-495a-b50d-42c41d6842fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:29.464031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.463932 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5p8\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-kube-api-access-mg5p8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.464031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.463980 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.464031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.463993 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.464031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.464005 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.464031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.464017 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.464031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.464029 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45ffb6e-c4bc-495a-b50d-42c41d6842fe-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:29.738904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.738816 4441 generic.go:356] "Generic (PLEG): container finished" podID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerID="d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1" exitCode=0 Nov 02 21:46:29.738904 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.738877 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" Nov 02 21:46:29.739132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.738899 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" event={"ID":"d45ffb6e-c4bc-495a-b50d-42c41d6842fe","Type":"ContainerDied","Data":"d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1"} Nov 02 21:46:29.739132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.738946 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz" event={"ID":"d45ffb6e-c4bc-495a-b50d-42c41d6842fe","Type":"ContainerDied","Data":"94285668477a0a149d065a21032cb725ab78b4882017bf4a23f6c4dee4ea3b37"} Nov 02 21:46:29.739132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.738998 4441 scope.go:122] "RemoveContainer" containerID="d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1" Nov 02 21:46:29.740508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.740486 4441 generic.go:356] "Generic (PLEG): container finished" podID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerID="014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb" exitCode=0 Nov 02 21:46:29.740632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.740536 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" Nov 02 21:46:29.740712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.740630 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" event={"ID":"cb843693-22ff-4e1b-8048-cb2b971339b4","Type":"ContainerDied","Data":"014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb"} Nov 02 21:46:29.740712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.740683 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7" event={"ID":"cb843693-22ff-4e1b-8048-cb2b971339b4","Type":"ContainerDied","Data":"d96e424fdb33e965723d6f67d57e2b20ef00b04fd908dd645641bb6ec1c86bc4"} Nov 02 21:46:29.750534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.750518 4441 scope.go:122] "RemoveContainer" containerID="d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1" Nov 02 21:46:29.750864 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:46:29.750837 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1\": container with ID starting with d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1 not found: ID does not exist" containerID="d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1" Nov 02 21:46:29.750993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.750874 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1"} err="failed to get container status \"d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1\": rpc error: code = NotFound desc = could not find container \"d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1\": container with ID starting with d55913548ac60737267fb1e52878296d515f3c80bec4b6646d758f238a86b1f1 not found: ID does not exist" Nov 02 21:46:29.750993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.750895 4441 scope.go:122] "RemoveContainer" containerID="014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb" Nov 02 21:46:29.761125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.761109 4441 scope.go:122] "RemoveContainer" containerID="014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb" Nov 02 21:46:29.761371 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:46:29.761354 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb\": container with ID starting with 014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb not found: ID does not exist" containerID="014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb" Nov 02 21:46:29.761410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.761378 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb"} err="failed to get container status \"014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb\": rpc error: code = NotFound desc = could not find container \"014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb\": container with ID starting with 014956bd34498376d0726f97275d9607d972cf6bc6629a5c1fb507df7983afcb not found: ID does not exist" Nov 02 21:46:29.897677 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.897651 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7"] Nov 02 21:46:29.902476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:29.902452 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qptp7"] Nov 02 21:46:30.005895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:30.005806 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz"] Nov 02 21:46:30.010790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:30.010764 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-djjtz"] Nov 02 21:46:30.603700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:30.603661 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" path="/var/lib/kubelet/pods/cb843693-22ff-4e1b-8048-cb2b971339b4/volumes" Nov 02 21:46:30.604426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:30.604398 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" path="/var/lib/kubelet/pods/d45ffb6e-c4bc-495a-b50d-42c41d6842fe/volumes" Nov 02 21:46:31.454128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:31.454093 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:35.157916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.157880 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj"] Nov 02 21:46:35.158794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.158764 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" Nov 02 21:46:35.158794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.158788 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" Nov 02 21:46:35.159019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.158831 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" Nov 02 21:46:35.159019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.158839 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" Nov 02 21:46:35.159145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.159064 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45ffb6e-c4bc-495a-b50d-42c41d6842fe" containerName="istio-proxy" Nov 02 21:46:35.159145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.159087 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb843693-22ff-4e1b-8048-cb2b971339b4" containerName="istio-proxy" Nov 02 21:46:35.164022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.164003 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.216614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.216583 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj"] Nov 02 21:46:35.257304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.257269 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:46:35.257304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.257303 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:46:35.257548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.257291 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:46:35.257548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.257360 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:46:35.344119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.344083 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.344316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.344262 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.344406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.344388 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmxz\" (UniqueName: \"kubernetes.io/projected/1bc4360d-0e1c-422f-b176-a4475691450c-kube-api-access-4zmxz\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.446089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.445992 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmxz\" (UniqueName: \"kubernetes.io/projected/1bc4360d-0e1c-422f-b176-a4475691450c-kube-api-access-4zmxz\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.446248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.446105 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.446321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.446290 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.446841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.446822 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.448613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.448593 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.499410 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.499381 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmxz\" (UniqueName: \"kubernetes.io/projected/1bc4360d-0e1c-422f-b176-a4475691450c-kube-api-access-4zmxz\") pod \"maas-api-revoke-sub-15edb8c5-w25mj\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.779782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.779694 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:35.957815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:35.957780 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:36.176232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:36.176200 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj"] Nov 02 21:46:36.178627 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:46:36.178595 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc4360d_0e1c_422f_b176_a4475691450c.slice/crio-beb08e51f78553328bd5092c81fa286ff6de748ca1b0f8ceac43b996332486b8 WatchSource:0}: Error finding container beb08e51f78553328bd5092c81fa286ff6de748ca1b0f8ceac43b996332486b8: Status 404 returned error can't find the container with id beb08e51f78553328bd5092c81fa286ff6de748ca1b0f8ceac43b996332486b8 Nov 02 21:46:36.812170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:36.812123 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" event={"ID":"1bc4360d-0e1c-422f-b176-a4475691450c","Type":"ContainerStarted","Data":"6c30da7ad5df625d272a0cf76792887dc4591a808fd75d759371b253b236c8cb"} Nov 02 21:46:36.812401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:36.812188 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" event={"ID":"1bc4360d-0e1c-422f-b176-a4475691450c","Type":"ContainerStarted","Data":"beb08e51f78553328bd5092c81fa286ff6de748ca1b0f8ceac43b996332486b8"} Nov 02 21:46:36.921197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:36.921087 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" podStartSLOduration=1.921067614 podStartE2EDuration="1.921067614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:36.918943286 +0000 UTC m=+2026.474571625" watchObservedRunningTime="2026-11-02 21:46:36.921067614 +0000 UTC m=+2026.476695951" Nov 02 21:46:37.824004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:37.823954 4441 generic.go:356] "Generic (PLEG): container finished" podID="1bc4360d-0e1c-422f-b176-a4475691450c" containerID="6c30da7ad5df625d272a0cf76792887dc4591a808fd75d759371b253b236c8cb" exitCode=0 Nov 02 21:46:37.824516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:37.824020 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" event={"ID":"1bc4360d-0e1c-422f-b176-a4475691450c","Type":"ContainerDied","Data":"6c30da7ad5df625d272a0cf76792887dc4591a808fd75d759371b253b236c8cb"} Nov 02 21:46:39.034240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.034214 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:39.070488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.070454 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-xks-ca\") pod \"1bc4360d-0e1c-422f-b176-a4475691450c\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " Nov 02 21:46:39.070683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.070665 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc4360d-0e1c-422f-b176-a4475691450c-kube-api-access-4zmxz\" (UniqueName: \"kubernetes.io/projected/1bc4360d-0e1c-422f-b176-a4475691450c-kube-api-access-4zmxz\") pod \"1bc4360d-0e1c-422f-b176-a4475691450c\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " Nov 02 21:46:39.070896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.070873 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-openshift-ca\") pod \"1bc4360d-0e1c-422f-b176-a4475691450c\" (UID: \"1bc4360d-0e1c-422f-b176-a4475691450c\") " Nov 02 21:46:39.071260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.071238 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-openshift-ca" pod "1bc4360d-0e1c-422f-b176-a4475691450c" (UID: "1bc4360d-0e1c-422f-b176-a4475691450c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:39.071512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.071490 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:39.072566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.072534 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-xks-ca" pod "1bc4360d-0e1c-422f-b176-a4475691450c" (UID: "1bc4360d-0e1c-422f-b176-a4475691450c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:39.073566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.073538 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc4360d-0e1c-422f-b176-a4475691450c-kube-api-access-4zmxz" pod "1bc4360d-0e1c-422f-b176-a4475691450c" (UID: "1bc4360d-0e1c-422f-b176-a4475691450c"). InnerVolumeSpecName "kube-api-access-4zmxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:39.173598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.173506 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc4360d-0e1c-422f-b176-a4475691450c-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:39.173598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.173545 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zmxz\" (UniqueName: \"kubernetes.io/projected/1bc4360d-0e1c-422f-b176-a4475691450c-kube-api-access-4zmxz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:39.842355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.842327 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" Nov 02 21:46:39.842529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.842358 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj" event={"ID":"1bc4360d-0e1c-422f-b176-a4475691450c","Type":"ContainerDied","Data":"beb08e51f78553328bd5092c81fa286ff6de748ca1b0f8ceac43b996332486b8"} Nov 02 21:46:39.842529 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:39.842400 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beb08e51f78553328bd5092c81fa286ff6de748ca1b0f8ceac43b996332486b8" Nov 02 21:46:40.498714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:40.498681 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:40.639261 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:46:40.639223 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:46:41.707725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.707685 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb"] Nov 02 21:46:41.708773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.708749 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1bc4360d-0e1c-422f-b176-a4475691450c" containerName="revoke-keys" Nov 02 21:46:41.708827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.708775 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc4360d-0e1c-422f-b176-a4475691450c" containerName="revoke-keys" Nov 02 21:46:41.708947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.708935 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc4360d-0e1c-422f-b176-a4475691450c" containerName="revoke-keys" Nov 02 21:46:41.713222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.713206 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.763304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.763275 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb"] Nov 02 21:46:41.805234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.805160 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:46:41.805234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.805213 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:46:41.805487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.805232 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:46:41.810300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.810276 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:46:41.811109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.811082 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62rc\" (UniqueName: \"kubernetes.io/projected/1b7cd0ac-1b01-4364-9353-864355548d46-kube-api-access-w62rc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.811217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.811150 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.811299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.811276 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.912796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.912767 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.912962 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.912941 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62rc\" (UniqueName: \"kubernetes.io/projected/1b7cd0ac-1b01-4364-9353-864355548d46-kube-api-access-w62rc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.913063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.913046 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.913632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.913609 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.915159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.915140 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:41.967202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:41.967147 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62rc\" (UniqueName: \"kubernetes.io/projected/1b7cd0ac-1b01-4364-9353-864355548d46-kube-api-access-w62rc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:42.028911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:42.028887 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:42.412716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:42.412686 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb"] Nov 02 21:46:42.417782 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:46:42.417747 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7cd0ac_1b01_4364_9353_864355548d46.slice/crio-c878fd3e85a72648d3febeb094cbc95cb263883bb987b9d7d0b990a797eea89c WatchSource:0}: Error finding container c878fd3e85a72648d3febeb094cbc95cb263883bb987b9d7d0b990a797eea89c: Status 404 returned error can't find the container with id c878fd3e85a72648d3febeb094cbc95cb263883bb987b9d7d0b990a797eea89c Nov 02 21:46:42.869669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:42.869635 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" event={"ID":"1b7cd0ac-1b01-4364-9353-864355548d46","Type":"ContainerStarted","Data":"880c86fe2c0eda6dce50a6ef76ac5f829dcade0cb45a416c089e3089d5538cbe"} Nov 02 21:46:42.870085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:42.869686 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" event={"ID":"1b7cd0ac-1b01-4364-9353-864355548d46","Type":"ContainerStarted","Data":"c878fd3e85a72648d3febeb094cbc95cb263883bb987b9d7d0b990a797eea89c"} Nov 02 21:46:42.982540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:42.982443 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" podStartSLOduration=1.9824286070000001 podStartE2EDuration="1.982428607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:42.97739519 +0000 UTC m=+2032.533023527" watchObservedRunningTime="2026-11-02 21:46:42.982428607 +0000 UTC m=+2032.538056945" Nov 02 21:46:43.879103 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:43.879070 4441 generic.go:356] "Generic (PLEG): container finished" podID="1b7cd0ac-1b01-4364-9353-864355548d46" containerID="880c86fe2c0eda6dce50a6ef76ac5f829dcade0cb45a416c089e3089d5538cbe" exitCode=0 Nov 02 21:46:43.879559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:43.879113 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" event={"ID":"1b7cd0ac-1b01-4364-9353-864355548d46","Type":"ContainerDied","Data":"880c86fe2c0eda6dce50a6ef76ac5f829dcade0cb45a416c089e3089d5538cbe"} Nov 02 21:46:45.072871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.072844 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:45.157443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.157359 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-openshift-ca\") pod \"1b7cd0ac-1b01-4364-9353-864355548d46\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " Nov 02 21:46:45.157443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.157402 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-xks-ca\") pod \"1b7cd0ac-1b01-4364-9353-864355548d46\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " Nov 02 21:46:45.157684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.157575 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7cd0ac-1b01-4364-9353-864355548d46-kube-api-access-w62rc\" (UniqueName: \"kubernetes.io/projected/1b7cd0ac-1b01-4364-9353-864355548d46-kube-api-access-w62rc\") pod \"1b7cd0ac-1b01-4364-9353-864355548d46\" (UID: \"1b7cd0ac-1b01-4364-9353-864355548d46\") " Nov 02 21:46:45.157802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.157776 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-openshift-ca" pod "1b7cd0ac-1b01-4364-9353-864355548d46" (UID: "1b7cd0ac-1b01-4364-9353-864355548d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:45.158171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.158151 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:45.159693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.159668 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7cd0ac-1b01-4364-9353-864355548d46-kube-api-access-w62rc" pod "1b7cd0ac-1b01-4364-9353-864355548d46" (UID: "1b7cd0ac-1b01-4364-9353-864355548d46"). InnerVolumeSpecName "kube-api-access-w62rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:45.159802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.159744 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-xks-ca" pod "1b7cd0ac-1b01-4364-9353-864355548d46" (UID: "1b7cd0ac-1b01-4364-9353-864355548d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:45.174391 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.174368 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs"] Nov 02 21:46:45.175048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.175030 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1b7cd0ac-1b01-4364-9353-864355548d46" containerName="revoke-keys" Nov 02 21:46:45.175135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.175049 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7cd0ac-1b01-4364-9353-864355548d46" containerName="revoke-keys" Nov 02 21:46:45.175282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.175268 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7cd0ac-1b01-4364-9353-864355548d46" containerName="revoke-keys" Nov 02 21:46:45.178767 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.178748 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.202706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.202626 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs"] Nov 02 21:46:45.259341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.259307 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.259485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.259391 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.259485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.259457 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkls\" (UniqueName: \"kubernetes.io/projected/e966f424-1925-4462-8e27-731ac98f4f81-kube-api-access-6hkls\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.259661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.259643 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w62rc\" (UniqueName: \"kubernetes.io/projected/1b7cd0ac-1b01-4364-9353-864355548d46-kube-api-access-w62rc\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:45.259697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.259665 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7cd0ac-1b01-4364-9353-864355548d46-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:45.328516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.328489 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:45.360502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.360476 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.360671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.360512 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkls\" (UniqueName: \"kubernetes.io/projected/e966f424-1925-4462-8e27-731ac98f4f81-kube-api-access-6hkls\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.360736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.360687 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.361345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.361324 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.362916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.362894 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.424085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.422106 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkls\" (UniqueName: \"kubernetes.io/projected/e966f424-1925-4462-8e27-731ac98f4f81-kube-api-access-6hkls\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.504314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.504285 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:45.872679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.872631 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:45.889258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.888957 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs"] Nov 02 21:46:45.892312 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:46:45.892277 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode966f424_1925_4462_8e27_731ac98f4f81.slice/crio-edf11da6e0d548efea90ad200b57c4772123342ae51be2155e9250057b35683e WatchSource:0}: Error finding container edf11da6e0d548efea90ad200b57c4772123342ae51be2155e9250057b35683e: Status 404 returned error can't find the container with id edf11da6e0d548efea90ad200b57c4772123342ae51be2155e9250057b35683e Nov 02 21:46:45.899696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.899670 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" Nov 02 21:46:45.899792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.899705 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb" event={"ID":"1b7cd0ac-1b01-4364-9353-864355548d46","Type":"ContainerDied","Data":"c878fd3e85a72648d3febeb094cbc95cb263883bb987b9d7d0b990a797eea89c"} Nov 02 21:46:45.899792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:45.899751 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c878fd3e85a72648d3febeb094cbc95cb263883bb987b9d7d0b990a797eea89c" Nov 02 21:46:46.912484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:46.912450 4441 generic.go:356] "Generic (PLEG): container finished" podID="e966f424-1925-4462-8e27-731ac98f4f81" containerID="c25c35a70700f7d63a3ec1d9712319947e8ca22b70ad27165ebffa17f59f56cc" exitCode=0 Nov 02 21:46:46.912889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:46.912529 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" event={"ID":"e966f424-1925-4462-8e27-731ac98f4f81","Type":"ContainerDied","Data":"c25c35a70700f7d63a3ec1d9712319947e8ca22b70ad27165ebffa17f59f56cc"} Nov 02 21:46:46.912889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:46.912588 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" event={"ID":"e966f424-1925-4462-8e27-731ac98f4f81","Type":"ContainerStarted","Data":"edf11da6e0d548efea90ad200b57c4772123342ae51be2155e9250057b35683e"} Nov 02 21:46:48.079010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.078984 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:48.202108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.202022 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-openshift-ca\") pod \"e966f424-1925-4462-8e27-731ac98f4f81\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " Nov 02 21:46:48.202108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.202087 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-xks-ca\") pod \"e966f424-1925-4462-8e27-731ac98f4f81\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " Nov 02 21:46:48.202403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.202270 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e966f424-1925-4462-8e27-731ac98f4f81-kube-api-access-6hkls\" (UniqueName: \"kubernetes.io/projected/e966f424-1925-4462-8e27-731ac98f4f81-kube-api-access-6hkls\") pod \"e966f424-1925-4462-8e27-731ac98f4f81\" (UID: \"e966f424-1925-4462-8e27-731ac98f4f81\") " Nov 02 21:46:48.202470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.202408 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-openshift-ca" pod "e966f424-1925-4462-8e27-731ac98f4f81" (UID: "e966f424-1925-4462-8e27-731ac98f4f81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:48.202855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.202835 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:48.204399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.204375 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-xks-ca" pod "e966f424-1925-4462-8e27-731ac98f4f81" (UID: "e966f424-1925-4462-8e27-731ac98f4f81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:48.207432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.207404 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e966f424-1925-4462-8e27-731ac98f4f81-kube-api-access-6hkls" pod "e966f424-1925-4462-8e27-731ac98f4f81" (UID: "e966f424-1925-4462-8e27-731ac98f4f81"). InnerVolumeSpecName "kube-api-access-6hkls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:48.304099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.304065 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e966f424-1925-4462-8e27-731ac98f4f81-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:48.304099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.304095 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkls\" (UniqueName: \"kubernetes.io/projected/e966f424-1925-4462-8e27-731ac98f4f81-kube-api-access-6hkls\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:48.933508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.933412 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" Nov 02 21:46:48.933508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.933436 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs" event={"ID":"e966f424-1925-4462-8e27-731ac98f4f81","Type":"ContainerDied","Data":"edf11da6e0d548efea90ad200b57c4772123342ae51be2155e9250057b35683e"} Nov 02 21:46:48.933508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:48.933491 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf11da6e0d548efea90ad200b57c4772123342ae51be2155e9250057b35683e" Nov 02 21:46:50.114261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:50.114229 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:51.572439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:51.572381 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:51.884260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:51.884150 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:53.346163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:53.346134 4441 scope.go:122] "RemoveContainer" containerID="bc6b4c69c82a52274f9271c0e7f9d45f02f986d948108032142be1e24a44304d" Nov 02 21:46:53.356790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:53.356771 4441 scope.go:122] "RemoveContainer" containerID="034bd5f54840006bf94b6dc833933815b66e0cb621967fce48f4dcb1772da218" Nov 02 21:46:53.445381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:53.445355 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:53.564673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:53.564599 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:55.548124 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:55.548080 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:57.992523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:57.992492 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:58.113654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:58.113615 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:59.788559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:46:59.788524 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:01.577669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:01.577634 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:01.929383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:01.929285 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:02.283049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:02.280839 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:02.589961 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:02.589925 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb"] Nov 02 21:47:02.632486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:02.632454 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-06366c-scgzb"] Nov 02 21:47:02.991810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:02.991693 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:04.585901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:04.585869 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339caa72-e44b-46a0-8e4d-500096385890" path="/var/lib/kubelet/pods/339caa72-e44b-46a0-8e4d-500096385890/volumes" Nov 02 21:47:05.089634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:05.089589 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:05.230442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:05.230407 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:05.425055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:05.416434 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:07.171214 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:07.171134 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:07.342001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:07.341218 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:47:07.378108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:07.378059 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:47:07.404154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:07.404111 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:47:07.422151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:07.422045 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:47:07.467202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:07.467035 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:47:07.488183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:07.486044 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:47:08.769679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:08.769644 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:08.906303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:08.906270 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:09.137197 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:09.137161 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg"] Nov 02 21:47:09.137406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:09.137369 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" containerID="cri-o://a9e5517f37ce79e9d17590b250f3e29ffce3d919c6c762116e5bbb639672cf03" gracePeriod=30 Nov 02 21:47:09.139586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:09.139562 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:10.001823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.001785 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm"] Nov 02 21:47:10.002489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.002472 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e966f424-1925-4462-8e27-731ac98f4f81" containerName="revoke-keys" Nov 02 21:47:10.002543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.002490 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e966f424-1925-4462-8e27-731ac98f4f81" containerName="revoke-keys" Nov 02 21:47:10.002619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.002609 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e966f424-1925-4462-8e27-731ac98f4f81" containerName="revoke-keys" Nov 02 21:47:10.006462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.006440 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.046765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.046740 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm"] Nov 02 21:47:10.056626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.056597 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:47:10.056789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.056677 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:47:10.056789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.056750 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:47:10.057149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.057127 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:47:10.158770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.158735 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.158982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.158891 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2k9d\" (UniqueName: \"kubernetes.io/projected/8bf91696-4238-4b18-8e51-54118ffaf65d-kube-api-access-f2k9d\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.159134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.159107 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.260576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.260480 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.260734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.260634 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.260734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.260702 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2k9d\" (UniqueName: \"kubernetes.io/projected/8bf91696-4238-4b18-8e51-54118ffaf65d-kube-api-access-f2k9d\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.261342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.261315 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.262908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.262887 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.314143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.314116 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2k9d\" (UniqueName: \"kubernetes.io/projected/8bf91696-4238-4b18-8e51-54118ffaf65d-kube-api-access-f2k9d\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.321010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.320982 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:10.629698 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:47:10.629662 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:47:10.736129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:10.736098 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm"] Nov 02 21:47:10.740717 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:47:10.740686 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf91696_4238_4b18_8e51_54118ffaf65d.slice/crio-7b2c783dac6c81c4ab0e7a586b1007584693ad07de657a9675fe00ddbd1a8156 WatchSource:0}: Error finding container 7b2c783dac6c81c4ab0e7a586b1007584693ad07de657a9675fe00ddbd1a8156: Status 404 returned error can't find the container with id 7b2c783dac6c81c4ab0e7a586b1007584693ad07de657a9675fe00ddbd1a8156 Nov 02 21:47:11.369012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:11.368952 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" event={"ID":"8bf91696-4238-4b18-8e51-54118ffaf65d","Type":"ContainerStarted","Data":"878af8762974757f1a7e930b2e75ab13d7080c56d31d8612c932f4dcec176193"} Nov 02 21:47:11.374091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:11.374052 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" event={"ID":"8bf91696-4238-4b18-8e51-54118ffaf65d","Type":"ContainerStarted","Data":"7b2c783dac6c81c4ab0e7a586b1007584693ad07de657a9675fe00ddbd1a8156"} Nov 02 21:47:11.504241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:11.504168 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" podStartSLOduration=2.5041507149999997 podStartE2EDuration="2.504150715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:11.475287847 +0000 UTC m=+2061.030916186" watchObservedRunningTime="2026-11-02 21:47:11.504150715 +0000 UTC m=+2061.059779048" Nov 02 21:47:12.236306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:12.236244 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:12.380430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:12.380389 4441 generic.go:356] "Generic (PLEG): container finished" podID="8bf91696-4238-4b18-8e51-54118ffaf65d" containerID="878af8762974757f1a7e930b2e75ab13d7080c56d31d8612c932f4dcec176193" exitCode=0 Nov 02 21:47:12.380933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:12.380639 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" event={"ID":"8bf91696-4238-4b18-8e51-54118ffaf65d","Type":"ContainerDied","Data":"878af8762974757f1a7e930b2e75ab13d7080c56d31d8612c932f4dcec176193"} Nov 02 21:47:12.662302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:12.662270 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:12.736216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:12.736177 4441 patch_prober.go:29] interesting pod/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:47:12.736417 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:12.736264 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:47:12.796280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:12.796249 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:13.609157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.609125 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:13.639288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.639256 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-xks-ca\") pod \"8bf91696-4238-4b18-8e51-54118ffaf65d\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " Nov 02 21:47:13.639518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.639502 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-openshift-ca\") pod \"8bf91696-4238-4b18-8e51-54118ffaf65d\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " Nov 02 21:47:13.639593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.639553 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf91696-4238-4b18-8e51-54118ffaf65d-kube-api-access-f2k9d\" (UniqueName: \"kubernetes.io/projected/8bf91696-4238-4b18-8e51-54118ffaf65d-kube-api-access-f2k9d\") pod \"8bf91696-4238-4b18-8e51-54118ffaf65d\" (UID: \"8bf91696-4238-4b18-8e51-54118ffaf65d\") " Nov 02 21:47:13.640234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.639903 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-openshift-ca" pod "8bf91696-4238-4b18-8e51-54118ffaf65d" (UID: "8bf91696-4238-4b18-8e51-54118ffaf65d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:47:13.640234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.640201 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:13.641496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.641468 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-xks-ca" pod "8bf91696-4238-4b18-8e51-54118ffaf65d" (UID: "8bf91696-4238-4b18-8e51-54118ffaf65d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:13.641960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.641940 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf91696-4238-4b18-8e51-54118ffaf65d-kube-api-access-f2k9d" pod "8bf91696-4238-4b18-8e51-54118ffaf65d" (UID: "8bf91696-4238-4b18-8e51-54118ffaf65d"). InnerVolumeSpecName "kube-api-access-f2k9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:13.741809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.741773 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2k9d\" (UniqueName: \"kubernetes.io/projected/8bf91696-4238-4b18-8e51-54118ffaf65d-kube-api-access-f2k9d\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:13.741809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:13.741810 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91696-4238-4b18-8e51-54118ffaf65d-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.409891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.409799 4441 generic.go:356] "Generic (PLEG): container finished" podID="367798ae-c8d3-444b-a221-99eec69e544c" containerID="a9e5517f37ce79e9d17590b250f3e29ffce3d919c6c762116e5bbb639672cf03" exitCode=0 Nov 02 21:47:14.409891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.409867 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" event={"ID":"367798ae-c8d3-444b-a221-99eec69e544c","Type":"ContainerDied","Data":"a9e5517f37ce79e9d17590b250f3e29ffce3d919c6c762116e5bbb639672cf03"} Nov 02 21:47:14.410167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.410039 4441 scope.go:122] "RemoveContainer" containerID="da38b0b3c6255f9ce62b3513a7183f12ae0e157eef368e2b9ae737f0a3dc1e60" Nov 02 21:47:14.413013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.412205 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" event={"ID":"8bf91696-4238-4b18-8e51-54118ffaf65d","Type":"ContainerDied","Data":"7b2c783dac6c81c4ab0e7a586b1007584693ad07de657a9675fe00ddbd1a8156"} Nov 02 21:47:14.413013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.412251 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b2c783dac6c81c4ab0e7a586b1007584693ad07de657a9675fe00ddbd1a8156" Nov 02 21:47:14.413013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.412319 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm" Nov 02 21:47:14.478855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.478823 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:47:14.568599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.568566 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367798ae-c8d3-444b-a221-99eec69e544c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367798ae-c8d3-444b-a221-99eec69e544c-istiod-ca-cert\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.568599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.568608 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-credential-socket\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.568790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.568629 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-certs\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.568790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.568750 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-kube-api-access-6sxfv\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-kube-api-access-6sxfv\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.568921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.568808 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-envoy\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.568921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.568859 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-socket\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.569082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.568944 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-istio-token\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-istio-token\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.569082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569023 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-certs" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:47:14.569082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569050 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367798ae-c8d3-444b-a221-99eec69e544c-istiod-ca-cert" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:47:14.569264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569158 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-data\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.569264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569190 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/367798ae-c8d3-444b-a221-99eec69e544c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367798ae-c8d3-444b-a221-99eec69e544c-istio-podinfo\") pod \"367798ae-c8d3-444b-a221-99eec69e544c\" (UID: \"367798ae-c8d3-444b-a221-99eec69e544c\") " Nov 02 21:47:14.569377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569325 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-credential-socket" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:47:14.569508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569470 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-data" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:47:14.569655 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569556 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-socket" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:47:14.569943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.569922 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.570177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.570161 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.570276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.570265 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367798ae-c8d3-444b-a221-99eec69e544c-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.570364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.570354 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.570449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.570438 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.571771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.571750 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-kube-api-access-6sxfv" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "kube-api-access-6sxfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:14.572072 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.572047 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-envoy" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:47:14.572257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.572227 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/367798ae-c8d3-444b-a221-99eec69e544c-istio-podinfo" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:47:14.572672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.572647 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-istio-token" pod "367798ae-c8d3-444b-a221-99eec69e544c" (UID: "367798ae-c8d3-444b-a221-99eec69e544c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:14.672676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.672598 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.672676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.672632 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367798ae-c8d3-444b-a221-99eec69e544c-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.672676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.672647 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sxfv\" (UniqueName: \"kubernetes.io/projected/367798ae-c8d3-444b-a221-99eec69e544c-kube-api-access-6sxfv\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:14.672676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:14.672662 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367798ae-c8d3-444b-a221-99eec69e544c-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:15.286113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:15.286071 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:47:15.286426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:15.286126 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:47:15.425928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:15.425901 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" Nov 02 21:47:15.426132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:15.425946 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg" event={"ID":"367798ae-c8d3-444b-a221-99eec69e544c","Type":"ContainerDied","Data":"126453da549dab897e67a0bff33c2e1b442f672fc9c6743735cb7aace6ecf3ca"} Nov 02 21:47:15.426132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:15.426039 4441 scope.go:122] "RemoveContainer" containerID="a9e5517f37ce79e9d17590b250f3e29ffce3d919c6c762116e5bbb639672cf03" Nov 02 21:47:15.586864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:15.586833 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg"] Nov 02 21:47:15.631697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:15.631648 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-06366c-openshift-default-6cd8c77f65-8swpg"] Nov 02 21:47:16.436186 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:16.436149 4441 generic.go:356] "Generic (PLEG): container finished" podID="22456806-d042-419a-b631-c348f4775b44" containerID="ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d" exitCode=0 Nov 02 21:47:16.436749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:16.436225 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" event={"ID":"22456806-d042-419a-b631-c348f4775b44","Type":"ContainerDied","Data":"ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d"} Nov 02 21:47:16.436749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:16.436719 4441 scope.go:122] "RemoveContainer" containerID="ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d" Nov 02 21:47:16.438443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:16.438418 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:47:16.438518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:16.438488 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:47:16.438557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:16.438521 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:47:16.591321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:16.591284 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367798ae-c8d3-444b-a221-99eec69e544c" path="/var/lib/kubelet/pods/367798ae-c8d3-444b-a221-99eec69e544c/volumes" Nov 02 21:47:17.165807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:17.165775 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:17.446595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:17.446511 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" event={"ID":"22456806-d042-419a-b631-c348f4775b44","Type":"ContainerStarted","Data":"635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773"} Nov 02 21:47:18.187992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:18.187342 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:47:18.190099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:18.190065 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 21:47:18.190307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:18.190281 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 21:47:19.183823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:19.183362 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 21:47:19.183823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:19.183433 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 21:47:20.184491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:20.184453 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 21:47:20.185142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:20.185108 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 21:47:21.184994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:21.183340 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 21:47:21.184994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:21.183400 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 21:47:21.443057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:21.441884 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:21.740066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:21.735021 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:21.854823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:21.854775 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:22.190374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:22.190333 4441 patch_prober.go:29] interesting pod/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 21:47:22.191015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:22.190956 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 21:47:23.114753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.114722 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:23.188216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.188182 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:47:23.188706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.188680 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:47:23.189452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.189434 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:47:23.557850 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.557823 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:23.769445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.769413 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:23.785805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.785776 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk"] Nov 02 21:47:23.790242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:23.790216 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-mm4fk"] Nov 02 21:47:24.585033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:24.585005 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b39b84-c2fa-4701-bb98-224e7fab65aa" path="/var/lib/kubelet/pods/c7b39b84-c2fa-4701-bb98-224e7fab65aa/volumes" Nov 02 21:47:27.821963 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:27.821925 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:28.568959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:28.568920 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:31.361271 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:31.361233 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:31.502327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:31.502283 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:33.427675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:33.427648 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:33.586091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:33.586060 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:33.767026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:33.766857 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:34.005399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.005359 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4"] Nov 02 21:47:34.006398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.006370 4441 cpu_manager.go:414] "containerMap: removing container" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" Nov 02 21:47:34.006585 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.006569 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" Nov 02 21:47:34.006926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.006902 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8bf91696-4238-4b18-8e51-54118ffaf65d" containerName="revoke-keys" Nov 02 21:47:34.006926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.006927 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf91696-4238-4b18-8e51-54118ffaf65d" containerName="revoke-keys" Nov 02 21:47:34.007122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.006956 4441 cpu_manager.go:414] "containerMap: removing container" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" Nov 02 21:47:34.007122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.006980 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" Nov 02 21:47:34.007232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.007217 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" Nov 02 21:47:34.007291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.007237 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf91696-4238-4b18-8e51-54118ffaf65d" containerName="revoke-keys" Nov 02 21:47:34.007291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.007249 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="367798ae-c8d3-444b-a221-99eec69e544c" containerName="istio-proxy" Nov 02 21:47:34.015897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.015870 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.062263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.061245 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4"] Nov 02 21:47:34.084991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.078225 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8knd\" (UniqueName: \"kubernetes.io/projected/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-kube-api-access-s8knd\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.084991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.078280 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.115996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.115266 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b983e02\"" Nov 02 21:47:34.115996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.115446 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b983e02-dockercfg-g72fd\"" Nov 02 21:47:34.224989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.223566 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8knd\" (UniqueName: \"kubernetes.io/projected/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-kube-api-access-s8knd\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.224989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.223623 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.266074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.266036 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x"] Nov 02 21:47:34.284537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.270510 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.284537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.274275 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x"] Nov 02 21:47:34.284537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.274391 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.290797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.290766 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8knd\" (UniqueName: \"kubernetes.io/projected/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-kube-api-access-s8knd\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.334106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.334073 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:34.430773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.430310 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7426317-bd74-4321-b8f6-9b31558ad3b1-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.430773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.430379 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzp8v\" (UniqueName: \"kubernetes.io/projected/d7426317-bd74-4321-b8f6-9b31558ad3b1-kube-api-access-nzp8v\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.531555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.531522 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzp8v\" (UniqueName: \"kubernetes.io/projected/d7426317-bd74-4321-b8f6-9b31558ad3b1-kube-api-access-nzp8v\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.531786 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.531772 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7426317-bd74-4321-b8f6-9b31558ad3b1-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.532460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.532439 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7426317-bd74-4321-b8f6-9b31558ad3b1-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.585751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.585722 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzp8v\" (UniqueName: \"kubernetes.io/projected/d7426317-bd74-4321-b8f6-9b31558ad3b1-kube-api-access-nzp8v\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.623854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.623827 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:34.725863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:34.725839 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4"] Nov 02 21:47:34.728357 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:47:34.728329 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7f6c6f_e22f_4319_9407_c5dea7ee55cd.slice/crio-2880cbaa1239c03993e2292fa6d2b04027efd10fd33261bf7213d61e0bf65e97 WatchSource:0}: Error finding container 2880cbaa1239c03993e2292fa6d2b04027efd10fd33261bf7213d61e0bf65e97: Status 404 returned error can't find the container with id 2880cbaa1239c03993e2292fa6d2b04027efd10fd33261bf7213d61e0bf65e97 Nov 02 21:47:35.010022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.009918 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x"] Nov 02 21:47:35.014158 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:47:35.014129 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7426317_bd74_4321_b8f6_9b31558ad3b1.slice/crio-455362f2a3e5cc5782c0e54874a30e0a57764c0412a4d0b95a7c70d6fd21302d WatchSource:0}: Error finding container 455362f2a3e5cc5782c0e54874a30e0a57764c0412a4d0b95a7c70d6fd21302d: Status 404 returned error can't find the container with id 455362f2a3e5cc5782c0e54874a30e0a57764c0412a4d0b95a7c70d6fd21302d Nov 02 21:47:35.675239 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.675198 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" event={"ID":"d7426317-bd74-4321-b8f6-9b31558ad3b1","Type":"ContainerStarted","Data":"f387cffb88f18a96bff96fcb6807cce1a3fbaeeb47831d3a9ea05a42c95c0155"} Nov 02 21:47:35.675678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.675255 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" event={"ID":"d7426317-bd74-4321-b8f6-9b31558ad3b1","Type":"ContainerStarted","Data":"455362f2a3e5cc5782c0e54874a30e0a57764c0412a4d0b95a7c70d6fd21302d"} Nov 02 21:47:35.675678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.675313 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:35.677113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.677083 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" event={"ID":"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd","Type":"ContainerStarted","Data":"1b1f1f7917994b611cbc4d3272ff9c14f72d5bb1915c02657d1e9d162075357d"} Nov 02 21:47:35.677263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.677131 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" event={"ID":"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd","Type":"ContainerStarted","Data":"2880cbaa1239c03993e2292fa6d2b04027efd10fd33261bf7213d61e0bf65e97"} Nov 02 21:47:35.677263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.677175 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:35.786734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.786681 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" podStartSLOduration=1.786665573 podStartE2EDuration="1.786665573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:35.78207027 +0000 UTC m=+2085.337698629" watchObservedRunningTime="2026-11-02 21:47:35.786665573 +0000 UTC m=+2085.342293910" Nov 02 21:47:35.891715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:35.891560 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" podStartSLOduration=2.891541509 podStartE2EDuration="2.891541509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:35.888852943 +0000 UTC m=+2085.444481281" watchObservedRunningTime="2026-11-02 21:47:35.891541509 +0000 UTC m=+2085.447169847" Nov 02 21:47:38.479312 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:38.479276 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:40.616777 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:47:40.616731 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:47:41.437088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:41.437055 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:41.557704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:41.557676 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:43.037812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:43.037770 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:46.688607 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:46.688565 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:47:46.689015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:46.688652 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:47:47.431985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:47.431942 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:48.094518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:48.094478 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:51.038567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:51.038538 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:47:51.039018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:51.039003 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:47:51.046195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:51.046170 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:47:51.046400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:51.046381 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:47:51.056914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:51.056897 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:47:51.056914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:51.056908 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:47:51.920449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:51.920417 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:52.773891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:52.773861 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:56.382088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:47:56.382051 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:00.421184 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:00.421152 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:01.824215 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:01.824173 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:01.955527 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:01.955431 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:02.086152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:02.086062 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:02.221473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:02.221440 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:02.432279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:02.432189 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb"] Nov 02 21:48:02.479240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:02.479204 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9f77b7-mt4qb"] Nov 02 21:48:02.585702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:02.585669 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7cd0ac-1b01-4364-9353-864355548d46" path="/var/lib/kubelet/pods/1b7cd0ac-1b01-4364-9353-864355548d46/volumes" Nov 02 21:48:02.770511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:02.770427 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:06.300317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:06.300258 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:07.887732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:07.887696 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:08.140556 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.140449 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:08.360280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.360239 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:08.383787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.383749 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs"] Nov 02 21:48:08.388233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.388208 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4d045-sdfrs"] Nov 02 21:48:08.493211 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.493116 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:08.588279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.588242 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e966f424-1925-4462-8e27-731ac98f4f81" path="/var/lib/kubelet/pods/e966f424-1925-4462-8e27-731ac98f4f81/volumes" Nov 02 21:48:08.761722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.761646 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:08.897207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:08.897174 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:09.585091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.585048 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh"] Nov 02 21:48:09.590899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.590875 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.641281 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.641252 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh"] Nov 02 21:48:09.672150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.672115 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.672314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.672258 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.672314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.672296 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kfqg\" (UniqueName: \"kubernetes.io/projected/97732e1f-b89c-453a-be74-0ddfc00d54da-kube-api-access-4kfqg\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.682928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.682900 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:48:09.683079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.682900 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:48:09.687361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.687343 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:48:09.687514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.687502 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:48:09.774301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.774270 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.774427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.774392 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.774501 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.774426 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfqg\" (UniqueName: \"kubernetes.io/projected/97732e1f-b89c-453a-be74-0ddfc00d54da-kube-api-access-4kfqg\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.775059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.775037 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.776696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.776669 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.827751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.827722 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfqg\" (UniqueName: \"kubernetes.io/projected/97732e1f-b89c-453a-be74-0ddfc00d54da-kube-api-access-4kfqg\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:09.905125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:09.905049 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:10.286926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:10.286900 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh"] Nov 02 21:48:10.289816 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:48:10.289781 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97732e1f_b89c_453a_be74_0ddfc00d54da.slice/crio-b12c52216e487bf8a4c5964cb6882241ed912444eddc6edea4c7f521e04b1118 WatchSource:0}: Error finding container b12c52216e487bf8a4c5964cb6882241ed912444eddc6edea4c7f521e04b1118: Status 404 returned error can't find the container with id b12c52216e487bf8a4c5964cb6882241ed912444eddc6edea4c7f521e04b1118 Nov 02 21:48:10.617315 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:48:10.617276 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:48:11.011020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:11.010940 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" event={"ID":"97732e1f-b89c-453a-be74-0ddfc00d54da","Type":"ContainerStarted","Data":"940baa1c365570205a8774b474278f9d221a52ea0a38846ceaddf05b6292ba13"} Nov 02 21:48:11.011020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:11.011012 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" event={"ID":"97732e1f-b89c-453a-be74-0ddfc00d54da","Type":"ContainerStarted","Data":"b12c52216e487bf8a4c5964cb6882241ed912444eddc6edea4c7f521e04b1118"} Nov 02 21:48:11.119118 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:11.119066 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" podStartSLOduration=2.119048086 podStartE2EDuration="2.119048086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:11.11792799 +0000 UTC m=+2120.673556328" watchObservedRunningTime="2026-11-02 21:48:11.119048086 +0000 UTC m=+2120.674676466" Nov 02 21:48:12.026566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:12.026530 4441 generic.go:356] "Generic (PLEG): container finished" podID="97732e1f-b89c-453a-be74-0ddfc00d54da" containerID="940baa1c365570205a8774b474278f9d221a52ea0a38846ceaddf05b6292ba13" exitCode=0 Nov 02 21:48:12.026958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:12.026618 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" event={"ID":"97732e1f-b89c-453a-be74-0ddfc00d54da","Type":"ContainerDied","Data":"940baa1c365570205a8774b474278f9d221a52ea0a38846ceaddf05b6292ba13"} Nov 02 21:48:13.103406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.103331 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:13.285178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.285113 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:13.435884 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.435851 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-xks-ca\") pod \"97732e1f-b89c-453a-be74-0ddfc00d54da\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " Nov 02 21:48:13.436133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.435910 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97732e1f-b89c-453a-be74-0ddfc00d54da-kube-api-access-4kfqg\" (UniqueName: \"kubernetes.io/projected/97732e1f-b89c-453a-be74-0ddfc00d54da-kube-api-access-4kfqg\") pod \"97732e1f-b89c-453a-be74-0ddfc00d54da\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " Nov 02 21:48:13.436133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.435947 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-openshift-ca\") pod \"97732e1f-b89c-453a-be74-0ddfc00d54da\" (UID: \"97732e1f-b89c-453a-be74-0ddfc00d54da\") " Nov 02 21:48:13.436397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.436372 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-openshift-ca" pod "97732e1f-b89c-453a-be74-0ddfc00d54da" (UID: "97732e1f-b89c-453a-be74-0ddfc00d54da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:13.436652 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.436629 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:13.438094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.438066 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-xks-ca" pod "97732e1f-b89c-453a-be74-0ddfc00d54da" (UID: "97732e1f-b89c-453a-be74-0ddfc00d54da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:13.438621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.438603 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97732e1f-b89c-453a-be74-0ddfc00d54da-kube-api-access-4kfqg" pod "97732e1f-b89c-453a-be74-0ddfc00d54da" (UID: "97732e1f-b89c-453a-be74-0ddfc00d54da"). InnerVolumeSpecName "kube-api-access-4kfqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:13.538329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.538257 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97732e1f-b89c-453a-be74-0ddfc00d54da-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:13.538329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:13.538285 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kfqg\" (UniqueName: \"kubernetes.io/projected/97732e1f-b89c-453a-be74-0ddfc00d54da-kube-api-access-4kfqg\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.044274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:14.044243 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" event={"ID":"97732e1f-b89c-453a-be74-0ddfc00d54da","Type":"ContainerDied","Data":"b12c52216e487bf8a4c5964cb6882241ed912444eddc6edea4c7f521e04b1118"} Nov 02 21:48:14.044274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:14.044270 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh" Nov 02 21:48:14.044504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:14.044296 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12c52216e487bf8a4c5964cb6882241ed912444eddc6edea4c7f521e04b1118" Nov 02 21:48:14.429019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:14.428985 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:14.562003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:14.561938 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:14.759297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:14.759193 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:15.555560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.555527 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj"] Nov 02 21:48:15.556246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.556231 4441 cpu_manager.go:414] "containerMap: removing container" podUID="97732e1f-b89c-453a-be74-0ddfc00d54da" containerName="revoke-keys" Nov 02 21:48:15.556297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.556247 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97732e1f-b89c-453a-be74-0ddfc00d54da" containerName="revoke-keys" Nov 02 21:48:15.556352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.556343 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="97732e1f-b89c-453a-be74-0ddfc00d54da" containerName="revoke-keys" Nov 02 21:48:15.559976 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.559948 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.615485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.615452 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj"] Nov 02 21:48:15.655114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.655071 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:48:15.655114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.655077 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:48:15.655358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.655123 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:48:15.656792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.656769 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:48:15.668338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.668315 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.668462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.668363 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwvt\" (UniqueName: \"kubernetes.io/projected/203ca872-2648-4c34-b299-8f0c49a4e80e-kube-api-access-4lwvt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.668660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.668641 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.770817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.770783 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.770960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.770850 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwvt\" (UniqueName: \"kubernetes.io/projected/203ca872-2648-4c34-b299-8f0c49a4e80e-kube-api-access-4lwvt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.771047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.770956 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.771564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.771543 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.773194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.773174 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.824178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.824147 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwvt\" (UniqueName: \"kubernetes.io/projected/203ca872-2648-4c34-b299-8f0c49a4e80e-kube-api-access-4lwvt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:15.874280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:15.874253 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:16.258432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:16.258356 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj"] Nov 02 21:48:16.262354 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:48:16.262319 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203ca872_2648_4c34_b299_8f0c49a4e80e.slice/crio-16c191b203b14ce516f814a23df5b7cd6d6b446677a3a245b89be7169ffa7f12 WatchSource:0}: Error finding container 16c191b203b14ce516f814a23df5b7cd6d6b446677a3a245b89be7169ffa7f12: Status 404 returned error can't find the container with id 16c191b203b14ce516f814a23df5b7cd6d6b446677a3a245b89be7169ffa7f12 Nov 02 21:48:16.345649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:16.345622 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:16.856420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:16.856352 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx"] Nov 02 21:48:16.860773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:16.860756 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:16.929820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:16.929789 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:16.935276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:16.935247 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7cc1b1-openshift-default-dockercfg-tqtp4\"" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012308 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-socket\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012424 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-data\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012463 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-credential-socket\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012508 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da32a5-06a1-46e4-8379-154d9524ad4d-istiod-ca-cert\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012644 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-envoy\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012671 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-certs\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012724 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-podinfo\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012779 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfvnj\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-kube-api-access-dfvnj\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.050554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.012833 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-token\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.106626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.106532 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" event={"ID":"203ca872-2648-4c34-b299-8f0c49a4e80e","Type":"ContainerStarted","Data":"8bd3dfc2193e8021007db8f9e6e8326a657e17493257731a217852691749b82c"} Nov 02 21:48:17.106883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.106856 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" event={"ID":"203ca872-2648-4c34-b299-8f0c49a4e80e","Type":"ContainerStarted","Data":"16c191b203b14ce516f814a23df5b7cd6d6b446677a3a245b89be7169ffa7f12"} Nov 02 21:48:17.114863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.114292 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvnj\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-kube-api-access-dfvnj\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.115113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.115064 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-token\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.115295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.115277 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-socket\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.115459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.115445 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-data\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.115576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.115562 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-credential-socket\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.115696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.115683 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da32a5-06a1-46e4-8379-154d9524ad4d-istiod-ca-cert\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.115889 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.115876 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-envoy\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.116002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.115988 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-certs\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.116140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.116125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-podinfo\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.117075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.117051 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-socket\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.119670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.119649 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da32a5-06a1-46e4-8379-154d9524ad4d-istiod-ca-cert\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.139841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.126272 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-data\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.140323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.126490 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-credential-socket\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.140467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.127211 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-certs\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.140561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.131788 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx"] Nov 02 21:48:17.140697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.137443 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-envoy\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.141570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.141545 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-podinfo\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.172991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.172942 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:17.223991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.200758 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvnj\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-kube-api-access-dfvnj\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.224257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.206567 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-token\") pod \"e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.479562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.479532 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:17.908540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.908486 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" podStartSLOduration=2.908469275 podStartE2EDuration="2.908469275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:17.243635481 +0000 UTC m=+2126.799263819" watchObservedRunningTime="2026-11-02 21:48:17.908469275 +0000 UTC m=+2127.464097684" Nov 02 21:48:17.911132 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.911107 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx"] Nov 02 21:48:17.914882 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:48:17.914859 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03da32a5_06a1_46e4_8379_154d9524ad4d.slice/crio-6bbe10f38ab01286a4d6d0f06f059a7e86d5f2c1501f8219071d145fe473c472 WatchSource:0}: Error finding container 6bbe10f38ab01286a4d6d0f06f059a7e86d5f2c1501f8219071d145fe473c472: Status 404 returned error can't find the container with id 6bbe10f38ab01286a4d6d0f06f059a7e86d5f2c1501f8219071d145fe473c472 Nov 02 21:48:17.917745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.917710 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:48:17.917827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.917777 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:48:17.917827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:17.917808 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:48:18.117946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.117901 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" event={"ID":"03da32a5-06a1-46e4-8379-154d9524ad4d","Type":"ContainerStarted","Data":"921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549"} Nov 02 21:48:18.118154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.117983 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" event={"ID":"03da32a5-06a1-46e4-8379-154d9524ad4d","Type":"ContainerStarted","Data":"6bbe10f38ab01286a4d6d0f06f059a7e86d5f2c1501f8219071d145fe473c472"} Nov 02 21:48:18.119608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.119585 4441 generic.go:356] "Generic (PLEG): container finished" podID="203ca872-2648-4c34-b299-8f0c49a4e80e" containerID="8bd3dfc2193e8021007db8f9e6e8326a657e17493257731a217852691749b82c" exitCode=0 Nov 02 21:48:18.119734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.119654 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" event={"ID":"203ca872-2648-4c34-b299-8f0c49a4e80e","Type":"ContainerDied","Data":"8bd3dfc2193e8021007db8f9e6e8326a657e17493257731a217852691749b82c"} Nov 02 21:48:18.126839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.126819 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:18.248052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.247982 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" podStartSLOduration=2.247945554 podStartE2EDuration="2.247945554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:18.226928389 +0000 UTC m=+2127.782556739" watchObservedRunningTime="2026-11-02 21:48:18.247945554 +0000 UTC m=+2127.803573893" Nov 02 21:48:18.480843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.480750 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:18.487633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:18.487604 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:19.129688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.129295 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:19.130265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.130246 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:48:19.297929 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.297899 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:19.360048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.360015 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/203ca872-2648-4c34-b299-8f0c49a4e80e-kube-api-access-4lwvt\" (UniqueName: \"kubernetes.io/projected/203ca872-2648-4c34-b299-8f0c49a4e80e-kube-api-access-4lwvt\") pod \"203ca872-2648-4c34-b299-8f0c49a4e80e\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " Nov 02 21:48:19.360226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.360213 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-xks-ca\") pod \"203ca872-2648-4c34-b299-8f0c49a4e80e\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " Nov 02 21:48:19.360278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.360261 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-openshift-ca\") pod \"203ca872-2648-4c34-b299-8f0c49a4e80e\" (UID: \"203ca872-2648-4c34-b299-8f0c49a4e80e\") " Nov 02 21:48:19.360880 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.360788 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-openshift-ca" pod "203ca872-2648-4c34-b299-8f0c49a4e80e" (UID: "203ca872-2648-4c34-b299-8f0c49a4e80e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:19.362341 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.362314 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203ca872-2648-4c34-b299-8f0c49a4e80e-kube-api-access-4lwvt" pod "203ca872-2648-4c34-b299-8f0c49a4e80e" (UID: "203ca872-2648-4c34-b299-8f0c49a4e80e"). InnerVolumeSpecName "kube-api-access-4lwvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:19.362448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.362418 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-xks-ca" pod "203ca872-2648-4c34-b299-8f0c49a4e80e" (UID: "203ca872-2648-4c34-b299-8f0c49a4e80e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:19.461774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.461701 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lwvt\" (UniqueName: \"kubernetes.io/projected/203ca872-2648-4c34-b299-8f0c49a4e80e-kube-api-access-4lwvt\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:19.461774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.461729 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:19.461774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:19.461741 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203ca872-2648-4c34-b299-8f0c49a4e80e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:20.138908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:20.138876 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" Nov 02 21:48:20.139333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:20.138914 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj" event={"ID":"203ca872-2648-4c34-b299-8f0c49a4e80e","Type":"ContainerDied","Data":"16c191b203b14ce516f814a23df5b7cd6d6b446677a3a245b89be7169ffa7f12"} Nov 02 21:48:20.139333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:20.138987 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c191b203b14ce516f814a23df5b7cd6d6b446677a3a245b89be7169ffa7f12" Nov 02 21:48:21.026286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:21.026247 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:21.096651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:21.096622 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5"] Nov 02 21:48:21.275625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:21.275585 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv"] Nov 02 21:48:21.340302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:21.340263 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:21.472651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:21.472616 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:21.599814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:21.599730 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:21.724374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:21.724341 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:23.078544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:23.078504 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:24.439371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:24.439335 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh"] Nov 02 21:48:24.439895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:24.439527 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" containerID="cri-o://050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a" gracePeriod=30 Nov 02 21:48:24.840442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:24.840408 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:48:24.840645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:24.840498 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:48:24.869678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:24.869647 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:25.057685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:25.057650 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:25.979360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:25.979327 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8"] Nov 02 21:48:26.083630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.083582 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp"] Nov 02 21:48:26.097288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.097253 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" podUID="d11b0d69-980b-46c2-a718-a853071a39c6" containerName="payload-processing" containerID="cri-o://d7b6691399030695b1e07dcddeda2676e865218c7bf7148a992c54f973e07635" gracePeriod=25 Nov 02 21:48:26.230238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.230159 4441 generic.go:356] "Generic (PLEG): container finished" podID="d11b0d69-980b-46c2-a718-a853071a39c6" containerID="d7b6691399030695b1e07dcddeda2676e865218c7bf7148a992c54f973e07635" exitCode=0 Nov 02 21:48:26.230364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.230242 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" event={"ID":"d11b0d69-980b-46c2-a718-a853071a39c6","Type":"ContainerDied","Data":"d7b6691399030695b1e07dcddeda2676e865218c7bf7148a992c54f973e07635"} Nov 02 21:48:26.267297 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:48:26.267273 4441 logging.go:55] [core] [Channel #4790 SubChannel #4791]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.177:50052", ServerName: "10.133.0.177:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.177:50052: connect: connection refused" Nov 02 21:48:26.278996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.276450 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" podUID="f2061072-8d27-49c9-b192-965a7858eaa8" containerName="payload-pre-processing" containerID="cri-o://27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698" gracePeriod=25 Nov 02 21:48:26.311052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.310392 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:26.451007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.447574 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:26.569017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.568217 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:48:26.727413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.727381 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:48:26.729533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.729508 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11b0d69-980b-46c2-a718-a853071a39c6-kube-api-access-n4hgz\" (UniqueName: \"kubernetes.io/projected/d11b0d69-980b-46c2-a718-a853071a39c6-kube-api-access-n4hgz\") pod \"d11b0d69-980b-46c2-a718-a853071a39c6\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " Nov 02 21:48:26.729692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.729634 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11b0d69-980b-46c2-a718-a853071a39c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11b0d69-980b-46c2-a718-a853071a39c6-plugins-config-volume\") pod \"d11b0d69-980b-46c2-a718-a853071a39c6\" (UID: \"d11b0d69-980b-46c2-a718-a853071a39c6\") " Nov 02 21:48:26.730100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.730071 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11b0d69-980b-46c2-a718-a853071a39c6-plugins-config-volume" pod "d11b0d69-980b-46c2-a718-a853071a39c6" (UID: "d11b0d69-980b-46c2-a718-a853071a39c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:26.730425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.730403 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11b0d69-980b-46c2-a718-a853071a39c6-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.732263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.732200 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11b0d69-980b-46c2-a718-a853071a39c6-kube-api-access-n4hgz" pod "d11b0d69-980b-46c2-a718-a853071a39c6" (UID: "d11b0d69-980b-46c2-a718-a853071a39c6"). InnerVolumeSpecName "kube-api-access-n4hgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:26.831125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.831102 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2061072-8d27-49c9-b192-965a7858eaa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2061072-8d27-49c9-b192-965a7858eaa8-plugins-config-volume\") pod \"f2061072-8d27-49c9-b192-965a7858eaa8\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " Nov 02 21:48:26.831243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.831232 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2061072-8d27-49c9-b192-965a7858eaa8-kube-api-access-jls7w\" (UniqueName: \"kubernetes.io/projected/f2061072-8d27-49c9-b192-965a7858eaa8-kube-api-access-jls7w\") pod \"f2061072-8d27-49c9-b192-965a7858eaa8\" (UID: \"f2061072-8d27-49c9-b192-965a7858eaa8\") " Nov 02 21:48:26.831430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.831409 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2061072-8d27-49c9-b192-965a7858eaa8-plugins-config-volume" pod "f2061072-8d27-49c9-b192-965a7858eaa8" (UID: "f2061072-8d27-49c9-b192-965a7858eaa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:26.831708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.831693 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2061072-8d27-49c9-b192-965a7858eaa8-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.831764 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.831711 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hgz\" (UniqueName: \"kubernetes.io/projected/d11b0d69-980b-46c2-a718-a853071a39c6-kube-api-access-n4hgz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.833241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.833220 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2061072-8d27-49c9-b192-965a7858eaa8-kube-api-access-jls7w" pod "f2061072-8d27-49c9-b192-965a7858eaa8" (UID: "f2061072-8d27-49c9-b192-965a7858eaa8"). InnerVolumeSpecName "kube-api-access-jls7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:26.933500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:26.933468 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jls7w\" (UniqueName: \"kubernetes.io/projected/f2061072-8d27-49c9-b192-965a7858eaa8-kube-api-access-jls7w\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:27.239570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.239540 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" Nov 02 21:48:27.240028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.239572 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" event={"ID":"d11b0d69-980b-46c2-a718-a853071a39c6","Type":"ContainerDied","Data":"8f21f542efab5ad25e4e8e75c64bfe247ce34a1e8854bc5046a6e71da13cdf70"} Nov 02 21:48:27.240028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.239669 4441 scope.go:122] "RemoveContainer" containerID="d7b6691399030695b1e07dcddeda2676e865218c7bf7148a992c54f973e07635" Nov 02 21:48:27.241442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.241417 4441 generic.go:356] "Generic (PLEG): container finished" podID="f2061072-8d27-49c9-b192-965a7858eaa8" containerID="27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698" exitCode=0 Nov 02 21:48:27.241530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.241460 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" event={"ID":"f2061072-8d27-49c9-b192-965a7858eaa8","Type":"ContainerDied","Data":"27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698"} Nov 02 21:48:27.241530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.241468 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" Nov 02 21:48:27.241530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.241494 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv" event={"ID":"f2061072-8d27-49c9-b192-965a7858eaa8","Type":"ContainerDied","Data":"e71b721fe13e4637c461dfcfaf0309293fd397f268a20745f1b2fda7345e10a4"} Nov 02 21:48:27.251057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.251031 4441 scope.go:122] "RemoveContainer" containerID="27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698" Nov 02 21:48:27.260941 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.260919 4441 scope.go:122] "RemoveContainer" containerID="27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698" Nov 02 21:48:27.261237 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:48:27.261208 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698\": container with ID starting with 27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698 not found: ID does not exist" containerID="27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698" Nov 02 21:48:27.261344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.261249 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698"} err="failed to get container status \"27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698\": rpc error: code = NotFound desc = could not find container \"27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698\": container with ID starting with 27e0111368e377146ceecc6f7a8ecc43acc11b6eb648b8f5ad3f1628fd668698 not found: ID does not exist" Nov 02 21:48:27.267441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.267416 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5" podUID="d11b0d69-980b-46c2-a718-a853071a39c6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.177:50052\" within 1s: context deadline exceeded" Nov 02 21:48:27.413055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.412704 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv"] Nov 02 21:48:27.428010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.426520 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5451a5-54b9b789c9-xs2rv"] Nov 02 21:48:27.542000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.541093 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5"] Nov 02 21:48:27.549000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:27.548921 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5451a5-566f6b7dbf-gmzc5"] Nov 02 21:48:28.451184 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:28.451140 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:28.588841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:28.588807 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11b0d69-980b-46c2-a718-a853071a39c6" path="/var/lib/kubelet/pods/d11b0d69-980b-46c2-a718-a853071a39c6/volumes" Nov 02 21:48:28.589294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:28.589278 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2061072-8d27-49c9-b192-965a7858eaa8" path="/var/lib/kubelet/pods/f2061072-8d27-49c9-b192-965a7858eaa8/volumes" Nov 02 21:48:29.738370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.738346 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:48:29.888338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888306 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-envoy\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888373 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-kube-api-access-jlzpp\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-kube-api-access-jlzpp\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888424 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-certs\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888449 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-data\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888548 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-credential-socket\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888591 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-token\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-token\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888626 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istiod-ca-cert\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888674 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-podinfo\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888710 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-socket\") pod \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\" (UID: \"cb19e54e-4bcb-4f51-b8c6-a1414e998ede\") " Nov 02 21:48:29.888920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.888757 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-certs" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.889364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.889339 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-credential-socket" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.889445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.889392 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.889505 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.889443 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-data" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.889505 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.889458 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istiod-ca-cert" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:29.889617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.889575 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-socket" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.891019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.890992 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-kube-api-access-jlzpp" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "kube-api-access-jlzpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:29.891136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.891059 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-envoy" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.891614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.891586 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-token" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:29.892069 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.892043 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-podinfo" pod "cb19e54e-4bcb-4f51-b8c6-a1414e998ede" (UID: "cb19e54e-4bcb-4f51-b8c6-a1414e998ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:48:29.991353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991267 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.991353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991297 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.991353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991310 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.991353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991323 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.991353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991335 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.991353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991349 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.991648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991362 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlzpp\" (UniqueName: \"kubernetes.io/projected/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-kube-api-access-jlzpp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.991648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:29.991376 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb19e54e-4bcb-4f51-b8c6-a1414e998ede-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:30.271382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.271280 4441 generic.go:356] "Generic (PLEG): container finished" podID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerID="050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a" exitCode=0 Nov 02 21:48:30.271382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.271344 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" Nov 02 21:48:30.271628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.271340 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" event={"ID":"cb19e54e-4bcb-4f51-b8c6-a1414e998ede","Type":"ContainerDied","Data":"050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a"} Nov 02 21:48:30.271628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.271493 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh" event={"ID":"cb19e54e-4bcb-4f51-b8c6-a1414e998ede","Type":"ContainerDied","Data":"55aed281397223432e283d6dd8b68f41fe37ab67b7c125967d174d9e5d784058"} Nov 02 21:48:30.271628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.271520 4441 scope.go:122] "RemoveContainer" containerID="050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a" Nov 02 21:48:30.284680 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.284662 4441 scope.go:122] "RemoveContainer" containerID="050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a" Nov 02 21:48:30.285135 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:48:30.285108 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a\": container with ID starting with 050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a not found: ID does not exist" containerID="050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a" Nov 02 21:48:30.285242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.285149 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a"} err="failed to get container status \"050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a\": rpc error: code = NotFound desc = could not find container \"050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a\": container with ID starting with 050a1aaa078f0e3e393820a03a15b9dfd1f561ccece1dec30bfe046c5f3d478a not found: ID does not exist" Nov 02 21:48:30.430026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.429992 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh"] Nov 02 21:48:30.479476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.479444 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-cf9jh"] Nov 02 21:48:30.588628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.588594 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" path="/var/lib/kubelet/pods/cb19e54e-4bcb-4f51-b8c6-a1414e998ede/volumes" Nov 02 21:48:30.980071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:30.979986 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" podUID="b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" containerName="payload-processing" containerID="cri-o://6e8938ae4e9f076f2c7a154a39185009611ef7273e9975e40cacf182f6c29fb4" gracePeriod=25 Nov 02 21:48:31.084619 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.084535 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" podUID="5c03d4a6-f46b-42bb-9c76-d3386304ed16" containerName="payload-pre-processing" containerID="cri-o://6e7399cc362cd05811872222afe7230379852e28ca775201ced738944c716c54" gracePeriod=25 Nov 02 21:48:31.284212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.284179 4441 generic.go:356] "Generic (PLEG): container finished" podID="b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" containerID="6e8938ae4e9f076f2c7a154a39185009611ef7273e9975e40cacf182f6c29fb4" exitCode=0 Nov 02 21:48:31.284384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.284341 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" event={"ID":"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9","Type":"ContainerDied","Data":"6e8938ae4e9f076f2c7a154a39185009611ef7273e9975e40cacf182f6c29fb4"} Nov 02 21:48:31.284464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.284394 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" event={"ID":"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9","Type":"ContainerDied","Data":"01511dd9c1858837c6d47d8c193d5efe8711d8f7ed6e2e5314ae251dc1a8c977"} Nov 02 21:48:31.284464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.284419 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01511dd9c1858837c6d47d8c193d5efe8711d8f7ed6e2e5314ae251dc1a8c977" Nov 02 21:48:31.286422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.286397 4441 generic.go:356] "Generic (PLEG): container finished" podID="5c03d4a6-f46b-42bb-9c76-d3386304ed16" containerID="6e7399cc362cd05811872222afe7230379852e28ca775201ced738944c716c54" exitCode=0 Nov 02 21:48:31.286550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.286473 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" event={"ID":"5c03d4a6-f46b-42bb-9c76-d3386304ed16","Type":"ContainerDied","Data":"6e7399cc362cd05811872222afe7230379852e28ca775201ced738944c716c54"} Nov 02 21:48:31.286793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.286774 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:48:31.315456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.315410 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-plugins-config-volume\") pod \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " Nov 02 21:48:31.315622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.315556 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-kube-api-access-4rphk\" (UniqueName: \"kubernetes.io/projected/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-kube-api-access-4rphk\") pod \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\" (UID: \"b2332601-0ea4-4830-8f4f-dd4a50e5ddc9\") " Nov 02 21:48:31.316862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.316833 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-plugins-config-volume" pod "b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" (UID: "b2332601-0ea4-4830-8f4f-dd4a50e5ddc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:31.322511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.322476 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-kube-api-access-4rphk" pod "b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" (UID: "b2332601-0ea4-4830-8f4f-dd4a50e5ddc9"). InnerVolumeSpecName "kube-api-access-4rphk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:31.395243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.395212 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:48:31.424855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.424820 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c03d4a6-f46b-42bb-9c76-d3386304ed16-kube-api-access-lzddg\" (UniqueName: \"kubernetes.io/projected/5c03d4a6-f46b-42bb-9c76-d3386304ed16-kube-api-access-lzddg\") pod \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " Nov 02 21:48:31.425091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.424878 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c03d4a6-f46b-42bb-9c76-d3386304ed16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c03d4a6-f46b-42bb-9c76-d3386304ed16-plugins-config-volume\") pod \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\" (UID: \"5c03d4a6-f46b-42bb-9c76-d3386304ed16\") " Nov 02 21:48:31.425597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.425572 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:31.425692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.425601 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rphk\" (UniqueName: \"kubernetes.io/projected/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9-kube-api-access-4rphk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:31.426027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.425983 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c03d4a6-f46b-42bb-9c76-d3386304ed16-plugins-config-volume" pod "5c03d4a6-f46b-42bb-9c76-d3386304ed16" (UID: "5c03d4a6-f46b-42bb-9c76-d3386304ed16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:31.428723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.428698 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c03d4a6-f46b-42bb-9c76-d3386304ed16-kube-api-access-lzddg" pod "5c03d4a6-f46b-42bb-9c76-d3386304ed16" (UID: "5c03d4a6-f46b-42bb-9c76-d3386304ed16"). InnerVolumeSpecName "kube-api-access-lzddg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:31.527109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.527025 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzddg\" (UniqueName: \"kubernetes.io/projected/5c03d4a6-f46b-42bb-9c76-d3386304ed16-kube-api-access-lzddg\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:31.527109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:31.527064 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c03d4a6-f46b-42bb-9c76-d3386304ed16-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:32.299692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.299605 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" Nov 02 21:48:32.299692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.299622 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp" event={"ID":"5c03d4a6-f46b-42bb-9c76-d3386304ed16","Type":"ContainerDied","Data":"4c2486984229e6f1d4ec05071df90ad94f2b61fe6843485f4d7bf80b2cedacad"} Nov 02 21:48:32.300228 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.299699 4441 scope.go:122] "RemoveContainer" containerID="6e7399cc362cd05811872222afe7230379852e28ca775201ced738944c716c54" Nov 02 21:48:32.300228 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.299904 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8" Nov 02 21:48:32.457218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.457179 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8"] Nov 02 21:48:32.507245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.507210 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-66c72e-66896d7d67-k7dq8"] Nov 02 21:48:32.564911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.564875 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp"] Nov 02 21:48:32.571953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.571917 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-66c72e-7cd79f45c5-68fdp"] Nov 02 21:48:32.588314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.588284 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c03d4a6-f46b-42bb-9c76-d3386304ed16" path="/var/lib/kubelet/pods/5c03d4a6-f46b-42bb-9c76-d3386304ed16/volumes" Nov 02 21:48:32.588751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.588738 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" path="/var/lib/kubelet/pods/b2332601-0ea4-4830-8f4f-dd4a50e5ddc9/volumes" Nov 02 21:48:32.882325 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:32.882236 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:34.482499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:34.482469 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:34.703978 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:34.703943 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:34.843221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:34.843126 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:34.963837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:34.963792 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:35.094810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:35.094712 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:35.097960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:35.097934 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm"] Nov 02 21:48:35.141290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:35.141263 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f13b18-29vzm"] Nov 02 21:48:35.466832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:35.466744 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:36.586891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:36.586857 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf91696-4238-4b18-8e51-54118ffaf65d" path="/var/lib/kubelet/pods/8bf91696-4238-4b18-8e51-54118ffaf65d/volumes" Nov 02 21:48:39.252144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:39.252108 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:40.495229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:40.495112 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:40.685904 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:48:40.685858 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:48:40.918437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:40.918402 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:41.230558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:41.230470 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:41.451076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:41.451038 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq"] Nov 02 21:48:41.536740 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:41.534394 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b7107c-w29zq"] Nov 02 21:48:41.751898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:41.751866 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:42.003101 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.003066 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:42.146015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.145963 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:42.281449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.281358 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:42.462625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.462590 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:42.588910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.588870 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79260734-7804-4c16-a20d-f7a67d726210" path="/var/lib/kubelet/pods/79260734-7804-4c16-a20d-f7a67d726210/volumes" Nov 02 21:48:42.590723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.590693 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:42.734024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.733984 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm"] Nov 02 21:48:42.780854 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:42.780825 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d3b375-kn6wm"] Nov 02 21:48:43.166094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:43.166060 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:43.305878 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:43.305844 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:44.584698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:44.584621 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a93ff5-f833-4558-9ae8-5befc8af8cc3" path="/var/lib/kubelet/pods/a9a93ff5-f833-4558-9ae8-5befc8af8cc3/volumes" Nov 02 21:48:47.429241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:47.429208 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:48.072581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:48.072531 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:48.272037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:48.272009 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:48.407333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:48.407236 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:48.415431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:48.415402 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg"] Nov 02 21:48:48.415671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:48.415643 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" containerID="cri-o://484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563" gracePeriod=30 Nov 02 21:48:48.891827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:48.891789 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:49.074989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.071506 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:49.234402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.234314 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:49.248998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.248938 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv"] Nov 02 21:48:49.249598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249583 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" Nov 02 21:48:49.249668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249598 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" Nov 02 21:48:49.249668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249622 4441 cpu_manager.go:414] "containerMap: removing container" podUID="5c03d4a6-f46b-42bb-9c76-d3386304ed16" containerName="payload-pre-processing" Nov 02 21:48:49.249668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249627 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c03d4a6-f46b-42bb-9c76-d3386304ed16" containerName="payload-pre-processing" Nov 02 21:48:49.249668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249647 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d11b0d69-980b-46c2-a718-a853071a39c6" containerName="payload-processing" Nov 02 21:48:49.249668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249652 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11b0d69-980b-46c2-a718-a853071a39c6" containerName="payload-processing" Nov 02 21:48:49.249668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249661 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f2061072-8d27-49c9-b192-965a7858eaa8" containerName="payload-pre-processing" Nov 02 21:48:49.249668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249667 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2061072-8d27-49c9-b192-965a7858eaa8" containerName="payload-pre-processing" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249678 4441 cpu_manager.go:414] "containerMap: removing container" podUID="203ca872-2648-4c34-b299-8f0c49a4e80e" containerName="revoke-keys" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249683 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203ca872-2648-4c34-b299-8f0c49a4e80e" containerName="revoke-keys" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249695 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" containerName="payload-processing" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249700 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" containerName="payload-processing" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249798 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11b0d69-980b-46c2-a718-a853071a39c6" containerName="payload-processing" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249807 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2061072-8d27-49c9-b192-965a7858eaa8" containerName="payload-pre-processing" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249816 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="203ca872-2648-4c34-b299-8f0c49a4e80e" containerName="revoke-keys" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249824 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2332601-0ea4-4830-8f4f-dd4a50e5ddc9" containerName="payload-processing" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249832 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb19e54e-4bcb-4f51-b8c6-a1414e998ede" containerName="istio-proxy" Nov 02 21:48:49.250154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.249840 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c03d4a6-f46b-42bb-9c76-d3386304ed16" containerName="payload-pre-processing" Nov 02 21:48:49.253325 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.253307 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.309082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.309049 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:48:49.313333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.313301 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:48:49.317665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.317640 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv"] Nov 02 21:48:49.332110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.332076 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.332501 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.332474 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xtx\" (UniqueName: \"kubernetes.io/projected/c4cb3598-70d6-404b-8255-545b453e1042-kube-api-access-24xtx\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.332662 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.332641 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.358301 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.358258 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:48:49.358518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.358368 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:48:49.492053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.491805 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xtx\" (UniqueName: \"kubernetes.io/projected/c4cb3598-70d6-404b-8255-545b453e1042-kube-api-access-24xtx\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.492053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.491862 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.492299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.492068 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.510172 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.493002 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.510789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.510758 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.547516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.547482 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xtx\" (UniqueName: \"kubernetes.io/projected/c4cb3598-70d6-404b-8255-545b453e1042-kube-api-access-24xtx\") pod \"maas-api-revoke-keys-e2e-api-b-32de96-jltwv\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.567170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.567145 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:49.942248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:49.942219 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv"] Nov 02 21:48:49.946564 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:48:49.946538 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cb3598_70d6_404b_8255_545b453e1042.slice/crio-9635341c050081491a5cfc31a3ed481bedb76b2ab6ebe74a88473f5bd375b1d2 WatchSource:0}: Error finding container 9635341c050081491a5cfc31a3ed481bedb76b2ab6ebe74a88473f5bd375b1d2: Status 404 returned error can't find the container with id 9635341c050081491a5cfc31a3ed481bedb76b2ab6ebe74a88473f5bd375b1d2 Nov 02 21:48:50.029682 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.029656 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg"] Nov 02 21:48:50.035236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.035217 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.086155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.086129 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg"] Nov 02 21:48:50.101514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.101471 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.101663 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.101557 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9bl9\" (UniqueName: \"kubernetes.io/projected/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-kube-api-access-q9bl9\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.101839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.101816 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.203525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.203463 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.203686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.203551 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.203686 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.203595 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9bl9\" (UniqueName: \"kubernetes.io/projected/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-kube-api-access-q9bl9\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.204440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.204416 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.269704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.269673 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.269704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.269701 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9bl9\" (UniqueName: \"kubernetes.io/projected/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-kube-api-access-q9bl9\") pod \"maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.349130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.349108 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:50.657573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.657532 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" event={"ID":"c4cb3598-70d6-404b-8255-545b453e1042","Type":"ContainerStarted","Data":"fc573bcef3d88a428a21b7e86688ff4060fcb492f97f58aa0749127d34d36710"} Nov 02 21:48:50.657768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.657593 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" event={"ID":"c4cb3598-70d6-404b-8255-545b453e1042","Type":"ContainerStarted","Data":"9635341c050081491a5cfc31a3ed481bedb76b2ab6ebe74a88473f5bd375b1d2"} Nov 02 21:48:50.721144 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.721115 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg"] Nov 02 21:48:50.726561 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:48:50.726534 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3589ef44_a9a7_4b81_b130_ecf06b31e1ad.slice/crio-7b7ae4803ee84a64ebcd956cb3946b1486add0a46978c08b21b09ca449f4c88d WatchSource:0}: Error finding container 7b7ae4803ee84a64ebcd956cb3946b1486add0a46978c08b21b09ca449f4c88d: Status 404 returned error can't find the container with id 7b7ae4803ee84a64ebcd956cb3946b1486add0a46978c08b21b09ca449f4c88d Nov 02 21:48:50.766640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:50.766597 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" podStartSLOduration=1.766584412 podStartE2EDuration="1.766584412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:50.764086228 +0000 UTC m=+2160.319714578" watchObservedRunningTime="2026-11-02 21:48:50.766584412 +0000 UTC m=+2160.322212748" Nov 02 21:48:51.669621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:51.669586 4441 generic.go:356] "Generic (PLEG): container finished" podID="c4cb3598-70d6-404b-8255-545b453e1042" containerID="fc573bcef3d88a428a21b7e86688ff4060fcb492f97f58aa0749127d34d36710" exitCode=0 Nov 02 21:48:51.670111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:51.669699 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" event={"ID":"c4cb3598-70d6-404b-8255-545b453e1042","Type":"ContainerDied","Data":"fc573bcef3d88a428a21b7e86688ff4060fcb492f97f58aa0749127d34d36710"} Nov 02 21:48:51.671664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:51.671637 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" event={"ID":"3589ef44-a9a7-4b81-b130-ecf06b31e1ad","Type":"ContainerStarted","Data":"446c068ac34ab436c3dab8637e45fa313510a250e8982fc5137361524f7a459e"} Nov 02 21:48:51.671785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:51.671685 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" event={"ID":"3589ef44-a9a7-4b81-b130-ecf06b31e1ad","Type":"ContainerStarted","Data":"7b7ae4803ee84a64ebcd956cb3946b1486add0a46978c08b21b09ca449f4c88d"} Nov 02 21:48:51.884987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:51.884858 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" podStartSLOduration=2.884843345 podStartE2EDuration="2.884843345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:51.884031846 +0000 UTC m=+2161.439660222" watchObservedRunningTime="2026-11-02 21:48:51.884843345 +0000 UTC m=+2161.440471680" Nov 02 21:48:52.538081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.538043 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:52.684084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.683956 4441 generic.go:356] "Generic (PLEG): container finished" podID="3589ef44-a9a7-4b81-b130-ecf06b31e1ad" containerID="446c068ac34ab436c3dab8637e45fa313510a250e8982fc5137361524f7a459e" exitCode=0 Nov 02 21:48:52.684084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.684049 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" event={"ID":"3589ef44-a9a7-4b81-b130-ecf06b31e1ad","Type":"ContainerDied","Data":"446c068ac34ab436c3dab8637e45fa313510a250e8982fc5137361524f7a459e"} Nov 02 21:48:52.705544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.705491 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:52.894316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.894289 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:52.939260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.939233 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4cb3598-70d6-404b-8255-545b453e1042-kube-api-access-24xtx\" (UniqueName: \"kubernetes.io/projected/c4cb3598-70d6-404b-8255-545b453e1042-kube-api-access-24xtx\") pod \"c4cb3598-70d6-404b-8255-545b453e1042\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " Nov 02 21:48:52.939462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.939443 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-openshift-ca\") pod \"c4cb3598-70d6-404b-8255-545b453e1042\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " Nov 02 21:48:52.939541 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.939526 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-xks-ca\") pod \"c4cb3598-70d6-404b-8255-545b453e1042\" (UID: \"c4cb3598-70d6-404b-8255-545b453e1042\") " Nov 02 21:48:52.939734 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.939707 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-openshift-ca" pod "c4cb3598-70d6-404b-8255-545b453e1042" (UID: "c4cb3598-70d6-404b-8255-545b453e1042"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:52.940136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.940114 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:52.941336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.941305 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cb3598-70d6-404b-8255-545b453e1042-kube-api-access-24xtx" pod "c4cb3598-70d6-404b-8255-545b453e1042" (UID: "c4cb3598-70d6-404b-8255-545b453e1042"). InnerVolumeSpecName "kube-api-access-24xtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:52.941674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:52.941644 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-xks-ca" pod "c4cb3598-70d6-404b-8255-545b453e1042" (UID: "c4cb3598-70d6-404b-8255-545b453e1042"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:53.041806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.041769 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cb3598-70d6-404b-8255-545b453e1042-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.041806 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.041802 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24xtx\" (UniqueName: \"kubernetes.io/projected/c4cb3598-70d6-404b-8255-545b453e1042-kube-api-access-24xtx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.685096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.683942 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:48:53.695632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.695595 4441 generic.go:356] "Generic (PLEG): container finished" podID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerID="484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563" exitCode=0 Nov 02 21:48:53.695754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.695646 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" Nov 02 21:48:53.695754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.695676 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" event={"ID":"f1ddcfe4-e1a2-4e98-adee-d30a1297583f","Type":"ContainerDied","Data":"484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563"} Nov 02 21:48:53.695754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.695744 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg" event={"ID":"f1ddcfe4-e1a2-4e98-adee-d30a1297583f","Type":"ContainerDied","Data":"efd0f09841656366894f30dc4a9b0fa22e5d9ba85ee34f9366465388c638afe8"} Nov 02 21:48:53.695926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.695771 4441 scope.go:122] "RemoveContainer" containerID="484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563" Nov 02 21:48:53.697448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.697429 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" Nov 02 21:48:53.697567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.697459 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv" event={"ID":"c4cb3598-70d6-404b-8255-545b453e1042","Type":"ContainerDied","Data":"9635341c050081491a5cfc31a3ed481bedb76b2ab6ebe74a88473f5bd375b1d2"} Nov 02 21:48:53.697567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.697503 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9635341c050081491a5cfc31a3ed481bedb76b2ab6ebe74a88473f5bd375b1d2" Nov 02 21:48:53.708194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.708174 4441 scope.go:122] "RemoveContainer" containerID="adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd" Nov 02 21:48:53.718508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.718473 4441 scope.go:122] "RemoveContainer" containerID="484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563" Nov 02 21:48:53.718792 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:48:53.718774 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563\": container with ID starting with 484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563 not found: ID does not exist" containerID="484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563" Nov 02 21:48:53.718865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.718800 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563"} err="failed to get container status \"484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563\": rpc error: code = NotFound desc = could not find container \"484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563\": container with ID starting with 484dece1f8a56d2ed3bd0ba14abaa2e991a081540c4780f1e53ee0d0564ab563 not found: ID does not exist" Nov 02 21:48:53.718865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.718818 4441 scope.go:122] "RemoveContainer" containerID="adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd" Nov 02 21:48:53.719077 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:48:53.719057 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd\": container with ID starting with adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd not found: ID does not exist" containerID="adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd" Nov 02 21:48:53.719145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.719085 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd"} err="failed to get container status \"adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd\": rpc error: code = NotFound desc = could not find container \"adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd\": container with ID starting with adc4443c1046c5eda822644380164a349c2e5b6498d3624ce1ff329a3d9236fd not found: ID does not exist" Nov 02 21:48:53.751060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751014 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-podinfo\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751076 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istiod-ca-cert\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751111 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-certs\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751342 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751253 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-credential-socket\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751384 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-data\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751437 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-envoy\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751525 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-socket\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751572 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-token\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751619 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-kube-api-access-ct5wl\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-kube-api-access-ct5wl\") pod \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\" (UID: \"f1ddcfe4-e1a2-4e98-adee-d30a1297583f\") " Nov 02 21:48:53.751699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751631 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istiod-ca-cert" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:53.751925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.751897 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-certs" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:53.752270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.752245 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.752368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.752278 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.752427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.752353 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-credential-socket" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:53.752481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.752438 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-socket" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:53.752537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.752488 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-data" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:53.753660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.753623 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-podinfo" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:48:53.754340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.754305 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-kube-api-access-ct5wl" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "kube-api-access-ct5wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:53.754469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.754437 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-token" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:53.754843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.754819 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-envoy" pod "f1ddcfe4-e1a2-4e98-adee-d30a1297583f" (UID: "f1ddcfe4-e1a2-4e98-adee-d30a1297583f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:53.853243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.853208 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.853243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.853237 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.853243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.853248 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.853564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.853257 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.853564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.853267 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct5wl\" (UniqueName: \"kubernetes.io/projected/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-kube-api-access-ct5wl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.853564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.853277 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.853564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.853287 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddcfe4-e1a2-4e98-adee-d30a1297583f-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.896896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.896871 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:53.954628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.954559 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-openshift-ca\") pod \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " Nov 02 21:48:53.954628 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.954625 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-xks-ca\") pod \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " Nov 02 21:48:53.954800 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.954703 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-kube-api-access-q9bl9\" (UniqueName: \"kubernetes.io/projected/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-kube-api-access-q9bl9\") pod \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\" (UID: \"3589ef44-a9a7-4b81-b130-ecf06b31e1ad\") " Nov 02 21:48:53.954932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.954913 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-openshift-ca" pod "3589ef44-a9a7-4b81-b130-ecf06b31e1ad" (UID: "3589ef44-a9a7-4b81-b130-ecf06b31e1ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:53.955273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.955253 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:53.956690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.956665 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-xks-ca" pod "3589ef44-a9a7-4b81-b130-ecf06b31e1ad" (UID: "3589ef44-a9a7-4b81-b130-ecf06b31e1ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:53.956824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:53.956808 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-kube-api-access-q9bl9" pod "3589ef44-a9a7-4b81-b130-ecf06b31e1ad" (UID: "3589ef44-a9a7-4b81-b130-ecf06b31e1ad"). InnerVolumeSpecName "kube-api-access-q9bl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:54.056957 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.056925 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:54.056957 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.056953 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9bl9\" (UniqueName: \"kubernetes.io/projected/3589ef44-a9a7-4b81-b130-ecf06b31e1ad-kube-api-access-q9bl9\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:54.115960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.115933 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg"] Nov 02 21:48:54.122596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.122566 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b7107c-openshift-default-778f76c545-8hgqg"] Nov 02 21:48:54.189651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.189608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:54.490683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.490652 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:54.585201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.585168 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" path="/var/lib/kubelet/pods/f1ddcfe4-e1a2-4e98-adee-d30a1297583f/volumes" Nov 02 21:48:54.708839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.708806 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" event={"ID":"3589ef44-a9a7-4b81-b130-ecf06b31e1ad","Type":"ContainerDied","Data":"7b7ae4803ee84a64ebcd956cb3946b1486add0a46978c08b21b09ca449f4c88d"} Nov 02 21:48:54.709321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.708853 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b7ae4803ee84a64ebcd956cb3946b1486add0a46978c08b21b09ca449f4c88d" Nov 02 21:48:54.709321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:54.708907 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg" Nov 02 21:48:57.233059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:48:57.233023 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:00.222273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:00.222222 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:00.347784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:00.347713 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:00.753595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:00.753562 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9"] Nov 02 21:49:00.991035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:00.991000 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:01.123212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:01.123178 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:01.747691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:01.747655 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:03.329500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:03.329468 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:03.449617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:03.449568 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:05.754047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:05.753997 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" podUID="3558ae06-97b2-4aa7-b9a2-93f7183ab76a" containerName="payload-pre-processing" containerID="cri-o://c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933" gracePeriod=25 Nov 02 21:49:06.066347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.066314 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:49:06.195099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.195045 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-kube-api-access-fjm8b\" (UniqueName: \"kubernetes.io/projected/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-kube-api-access-fjm8b\") pod \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " Nov 02 21:49:06.195294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.195145 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-plugins-config-volume\") pod \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\" (UID: \"3558ae06-97b2-4aa7-b9a2-93f7183ab76a\") " Nov 02 21:49:06.195887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.195856 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-plugins-config-volume" pod "3558ae06-97b2-4aa7-b9a2-93f7183ab76a" (UID: "3558ae06-97b2-4aa7-b9a2-93f7183ab76a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:06.201647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.200531 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-kube-api-access-fjm8b" pod "3558ae06-97b2-4aa7-b9a2-93f7183ab76a" (UID: "3558ae06-97b2-4aa7-b9a2-93f7183ab76a"). InnerVolumeSpecName "kube-api-access-fjm8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:06.297994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.297883 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjm8b\" (UniqueName: \"kubernetes.io/projected/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-kube-api-access-fjm8b\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:06.297994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.297917 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3558ae06-97b2-4aa7-b9a2-93f7183ab76a-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:06.332291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.332253 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:06.907042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.895070 4441 generic.go:356] "Generic (PLEG): container finished" podID="3558ae06-97b2-4aa7-b9a2-93f7183ab76a" containerID="c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933" exitCode=0 Nov 02 21:49:06.907042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.895380 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" event={"ID":"3558ae06-97b2-4aa7-b9a2-93f7183ab76a","Type":"ContainerDied","Data":"c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933"} Nov 02 21:49:06.907042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.895436 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" event={"ID":"3558ae06-97b2-4aa7-b9a2-93f7183ab76a","Type":"ContainerDied","Data":"7a6a7c25190a69cf71d5feae7c05f74b9dcc04b73b954caa21b4a74b368eb613"} Nov 02 21:49:06.907042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.895467 4441 scope.go:122] "RemoveContainer" containerID="c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933" Nov 02 21:49:06.907042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.895653 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9" Nov 02 21:49:06.945121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.936949 4441 scope.go:122] "RemoveContainer" containerID="c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933" Nov 02 21:49:06.945121 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:49:06.937875 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933\": container with ID starting with c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933 not found: ID does not exist" containerID="c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933" Nov 02 21:49:06.945121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:06.938249 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933"} err="failed to get container status \"c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933\": rpc error: code = NotFound desc = could not find container \"c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933\": container with ID starting with c03ce4fce1606c6a690e3233e19185197b4428520243c366c8ae75b1bde7f933 not found: ID does not exist" Nov 02 21:49:07.054320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:07.054286 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9"] Nov 02 21:49:07.100707 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:07.100675 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a1bf7d-6955977c5-wvmp9"] Nov 02 21:49:08.592189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:08.592155 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3558ae06-97b2-4aa7-b9a2-93f7183ab76a" path="/var/lib/kubelet/pods/3558ae06-97b2-4aa7-b9a2-93f7183ab76a/volumes" Nov 02 21:49:09.239825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:09.239789 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:09.369898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:09.369869 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:10.617705 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:49:10.617668 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:49:10.870906 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:10.870799 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:15.405087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:15.405051 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:15.863588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:15.863559 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:20.009310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:20.009254 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:21.643443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:21.643393 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:23.918218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:23.918148 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:25.423693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:25.423657 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:25.634373 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:25.634331 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:25.765284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:25.765178 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:25.912558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:25.912524 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh"] Nov 02 21:49:25.958868 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:25.958830 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5451a5-vvqzh"] Nov 02 21:49:26.312248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:26.312198 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:26.585457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:26.585371 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97732e1f-b89c-453a-be74-0ddfc00d54da" path="/var/lib/kubelet/pods/97732e1f-b89c-453a-be74-0ddfc00d54da/volumes" Nov 02 21:49:27.352324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.352292 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw"] Nov 02 21:49:27.352952 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.352935 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3558ae06-97b2-4aa7-b9a2-93f7183ab76a" containerName="payload-pre-processing" Nov 02 21:49:27.352952 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.352953 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3558ae06-97b2-4aa7-b9a2-93f7183ab76a" containerName="payload-pre-processing" Nov 02 21:49:27.353134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.352994 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" Nov 02 21:49:27.353134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353000 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" Nov 02 21:49:27.353134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353009 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c4cb3598-70d6-404b-8255-545b453e1042" containerName="revoke-keys" Nov 02 21:49:27.353134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353014 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cb3598-70d6-404b-8255-545b453e1042" containerName="revoke-keys" Nov 02 21:49:27.353134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353037 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3589ef44-a9a7-4b81-b130-ecf06b31e1ad" containerName="revoke-keys" Nov 02 21:49:27.353134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353042 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3589ef44-a9a7-4b81-b130-ecf06b31e1ad" containerName="revoke-keys" Nov 02 21:49:27.353444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353142 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" Nov 02 21:49:27.353444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353149 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cb3598-70d6-404b-8255-545b453e1042" containerName="revoke-keys" Nov 02 21:49:27.353444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353160 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" Nov 02 21:49:27.353444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353168 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3589ef44-a9a7-4b81-b130-ecf06b31e1ad" containerName="revoke-keys" Nov 02 21:49:27.353444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.353175 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3558ae06-97b2-4aa7-b9a2-93f7183ab76a" containerName="payload-pre-processing" Nov 02 21:49:27.357890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.357865 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.407192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.407167 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b983e02-openshift-default-dockercfg-xbqwp\"" Nov 02 21:49:27.410044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410020 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410068 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410195 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16840db4-8c59-4fbf-87a6-24a83d2b379b-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410279 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410330 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410347 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410405 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410447 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xklht\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-kube-api-access-xklht\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.410580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.410567 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.511998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.511934 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512014 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xklht\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-kube-api-access-xklht\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512160 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512274 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512279 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512332 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512479 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512459 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16840db4-8c59-4fbf-87a6-24a83d2b379b-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512551 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512584 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512612 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512639 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512583 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.512894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.512873 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.513119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.513102 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16840db4-8c59-4fbf-87a6-24a83d2b379b-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.514650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.514623 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.514755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.514689 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.545782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.545737 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw"] Nov 02 21:49:27.565574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.565544 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xklht\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-kube-api-access-xklht\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.566001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.565959 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:27.672068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:27.671953 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:28.090327 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:49:28.090292 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16840db4_8c59_4fbf_87a6_24a83d2b379b.slice/crio-4a33cd3fa8b2ccc91993705447163ff964618220cd294cddfb8c0424434ee423 WatchSource:0}: Error finding container 4a33cd3fa8b2ccc91993705447163ff964618220cd294cddfb8c0424434ee423: Status 404 returned error can't find the container with id 4a33cd3fa8b2ccc91993705447163ff964618220cd294cddfb8c0424434ee423 Nov 02 21:49:28.090576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:28.090550 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw"] Nov 02 21:49:28.093026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:28.092996 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:28.093119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:28.093057 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:28.093119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:28.093088 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:28.159277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:28.159233 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" event={"ID":"16840db4-8c59-4fbf-87a6-24a83d2b379b","Type":"ContainerStarted","Data":"4a33cd3fa8b2ccc91993705447163ff964618220cd294cddfb8c0424434ee423"} Nov 02 21:49:29.184708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:29.183397 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" event={"ID":"16840db4-8c59-4fbf-87a6-24a83d2b379b","Type":"ContainerStarted","Data":"2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065"} Nov 02 21:49:29.353365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:29.353301 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" podStartSLOduration=2.353287723 podStartE2EDuration="2.353287723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:29.302209867 +0000 UTC m=+2198.857838204" watchObservedRunningTime="2026-11-02 21:49:29.353287723 +0000 UTC m=+2198.908916059" Nov 02 21:49:29.684993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:29.682218 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:29.684993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:29.682342 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 21:49:29.684993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:29.682385 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 21:49:29.777993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:29.775355 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:30.676810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:30.676338 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 21:49:30.676810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:30.676411 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 21:49:31.515057 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:31.515026 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:31.657400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:31.657368 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:31.689991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:31.687290 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 21:49:31.689991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:31.687347 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 21:49:31.794732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:31.794646 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:31.925037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:31.924292 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:31.963863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:31.963832 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj"] Nov 02 21:49:32.011110 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.011078 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-66c72e-x2jqj"] Nov 02 21:49:32.347254 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.347224 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:32.485613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.485581 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:32.604357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.604236 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="203ca872-2648-4c34-b299-8f0c49a4e80e" path="/var/lib/kubelet/pods/203ca872-2648-4c34-b299-8f0c49a4e80e/volumes" Nov 02 21:49:32.618104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.618075 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:32.676992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.674297 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 21:49:32.676992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.674352 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 21:49:32.830847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.830799 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:32.945657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.945550 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx"] Nov 02 21:49:32.945883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.945847 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" containerID="cri-o://18ee7b2c45e544d11ad421925f784af0cff36de74bf7dbde61d18c4be41c28ae" gracePeriod=30 Nov 02 21:49:32.949893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.949864 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8"] Nov 02 21:49:32.950277 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:32.950249 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" containerID="cri-o://15b03cbef65bd4774e36c29ee2f515fd082dc277d2cfaf576c1358f6076b569d" gracePeriod=30 Nov 02 21:49:33.071912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:33.071869 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj"] Nov 02 21:49:33.127443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:33.127397 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" containerID="cri-o://44c8ba81ed8c48da7c5d5c41c0e546740516da4e62ea2c8d9c8c66b57e4406ab" gracePeriod=30 Nov 02 21:49:33.657261 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:33.657228 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:33.674991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:33.674292 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Nov 02 21:49:33.674991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:33.674347 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Nov 02 21:49:34.395026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:34.368462 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:34.516244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:34.516204 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:34.639656 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:34.639597 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:34.839298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:34.839268 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:34.839679 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:34.839645 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:34.840962 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:34.840940 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:49:37.236530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:37.236470 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:38.255311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.254895 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:38.388456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.388384 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:38.467000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.466615 4441 generic.go:356] "Generic (PLEG): container finished" podID="503d1f35-2825-4893-aacb-3070b61de20b" containerID="15b03cbef65bd4774e36c29ee2f515fd082dc277d2cfaf576c1358f6076b569d" exitCode=0 Nov 02 21:49:38.467000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.466870 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" event={"ID":"503d1f35-2825-4893-aacb-3070b61de20b","Type":"ContainerDied","Data":"15b03cbef65bd4774e36c29ee2f515fd082dc277d2cfaf576c1358f6076b569d"} Nov 02 21:49:38.467000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.466924 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" event={"ID":"503d1f35-2825-4893-aacb-3070b61de20b","Type":"ContainerDied","Data":"d4fbcdec61a6fca2e799d1526dc46157a0c969bb45e0e3ca045e5ceee515cfc8"} Nov 02 21:49:38.467000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.466949 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4fbcdec61a6fca2e799d1526dc46157a0c969bb45e0e3ca045e5ceee515cfc8" Nov 02 21:49:38.470911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.470756 4441 generic.go:356] "Generic (PLEG): container finished" podID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerID="44c8ba81ed8c48da7c5d5c41c0e546740516da4e62ea2c8d9c8c66b57e4406ab" exitCode=0 Nov 02 21:49:38.470911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.470852 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" event={"ID":"913f1206-00cc-4569-b38d-20897a9a2cd8","Type":"ContainerDied","Data":"44c8ba81ed8c48da7c5d5c41c0e546740516da4e62ea2c8d9c8c66b57e4406ab"} Nov 02 21:49:38.473128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.472961 4441 generic.go:356] "Generic (PLEG): container finished" podID="6722f506-00e1-4596-aaab-34e7a8da562b" containerID="18ee7b2c45e544d11ad421925f784af0cff36de74bf7dbde61d18c4be41c28ae" exitCode=0 Nov 02 21:49:38.473128 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.473082 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" event={"ID":"6722f506-00e1-4596-aaab-34e7a8da562b","Type":"ContainerDied","Data":"18ee7b2c45e544d11ad421925f784af0cff36de74bf7dbde61d18c4be41c28ae"} Nov 02 21:49:38.486068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.486009 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:49:38.512805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.510096 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:49:38.562954 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.562918 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-socket\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.563354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.563328 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-credential-socket\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.563514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.563498 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-envoy\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.563675 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.563660 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-podinfo\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.563851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.563821 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-kube-api-access-c2ssw\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-kube-api-access-c2ssw\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.564592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.564568 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-credential-socket\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.564827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.564793 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-data\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.565156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.565135 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503d1f35-2825-4893-aacb-3070b61de20b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503d1f35-2825-4893-aacb-3070b61de20b-istiod-ca-cert\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.565365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.565349 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-data\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.565525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.565509 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-envoy\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.565731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.565711 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/503d1f35-2825-4893-aacb-3070b61de20b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503d1f35-2825-4893-aacb-3070b61de20b-istio-podinfo\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.565921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.565892 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-istio-token\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-istio-token\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.566179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.566158 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-socket\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.566338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.566320 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-token\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.566551 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.566533 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913f1206-00cc-4569-b38d-20897a9a2cd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913f1206-00cc-4569-b38d-20897a9a2cd8-istiod-ca-cert\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567044 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-certs\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567113 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-kube-api-access-lv8b7\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-kube-api-access-lv8b7\") pod \"913f1206-00cc-4569-b38d-20897a9a2cd8\" (UID: \"913f1206-00cc-4569-b38d-20897a9a2cd8\") " Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567145 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-certs\") pod \"503d1f35-2825-4893-aacb-3070b61de20b\" (UID: \"503d1f35-2825-4893-aacb-3070b61de20b\") " Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567157 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-socket" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567181 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503d1f35-2825-4893-aacb-3070b61de20b-istiod-ca-cert" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567429 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-credential-socket" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567447 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-certs" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567452 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-data" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567562 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913f1206-00cc-4569-b38d-20897a9a2cd8-istiod-ca-cert" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567961 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.567984 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-socket" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568001 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913f1206-00cc-4569-b38d-20897a9a2cd8-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568018 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568031 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568045 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568059 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503d1f35-2825-4893-aacb-3070b61de20b-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568108 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-certs" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568118 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-data" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568220 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-credential-socket" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.568361 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.568335 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-kube-api-access-c2ssw" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "kube-api-access-c2ssw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.569559 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.569056 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-envoy" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.572133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.572107 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/503d1f35-2825-4893-aacb-3070b61de20b-istio-podinfo" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:38.572240 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.572224 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-envoy" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.573297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.573273 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-token" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.573398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.573342 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-istio-token" pod "503d1f35-2825-4893-aacb-3070b61de20b" (UID: "503d1f35-2825-4893-aacb-3070b61de20b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.575608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.575550 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-podinfo" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:38.575608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.575589 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-kube-api-access-lv8b7" pod "913f1206-00cc-4569-b38d-20897a9a2cd8" (UID: "913f1206-00cc-4569-b38d-20897a9a2cd8"). InnerVolumeSpecName "kube-api-access-lv8b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.589788 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.588746 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:38.611883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.602874 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9"] Nov 02 21:49:38.611883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.603116 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" containerID="cri-o://635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773" gracePeriod=30 Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.691920 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.691980 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8b7\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-kube-api-access-lv8b7\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692000 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692015 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692030 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692044 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692058 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ssw\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-kube-api-access-c2ssw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692071 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692086 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503d1f35-2825-4893-aacb-3070b61de20b-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692099 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503d1f35-2825-4893-aacb-3070b61de20b-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692113 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503d1f35-2825-4893-aacb-3070b61de20b-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.701015 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.692126 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913f1206-00cc-4569-b38d-20897a9a2cd8-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.846812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.846719 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:49:38.900444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.900409 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-istio-token\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-istio-token\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.901273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.901245 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-socket" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.901405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.901304 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-socket\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.901482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.901399 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-credential-socket\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.901647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.901628 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-credential-socket" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.901746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.901681 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-data\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.901908 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.901890 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-data" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.902056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.902029 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-certs\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.902291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.902275 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-certs" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.902395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.902342 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-kube-api-access-88vsf\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.902721 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.902704 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-envoy\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.903257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.902833 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6722f506-00e1-4596-aaab-34e7a8da562b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6722f506-00e1-4596-aaab-34e7a8da562b-istio-podinfo\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.903257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.902875 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6722f506-00e1-4596-aaab-34e7a8da562b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6722f506-00e1-4596-aaab-34e7a8da562b-istiod-ca-cert\") pod \"6722f506-00e1-4596-aaab-34e7a8da562b\" (UID: \"6722f506-00e1-4596-aaab-34e7a8da562b\") " Nov 02 21:49:38.903444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.903395 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6722f506-00e1-4596-aaab-34e7a8da562b-istiod-ca-cert" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:38.904265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.904243 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.904464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.904440 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6722f506-00e1-4596-aaab-34e7a8da562b-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.904614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.904599 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.904728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.904711 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.904828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.904813 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.905667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.905622 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-kube-api-access-88vsf" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "kube-api-access-88vsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.905787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.905736 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-istio-token" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.906005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.905945 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6722f506-00e1-4596-aaab-34e7a8da562b-istio-podinfo" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:38.907545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:38.907520 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-envoy" pod "6722f506-00e1-4596-aaab-34e7a8da562b" (UID: "6722f506-00e1-4596-aaab-34e7a8da562b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:39.006459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.006426 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6722f506-00e1-4596-aaab-34e7a8da562b-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:39.006459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.006459 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:39.006630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.006473 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/6722f506-00e1-4596-aaab-34e7a8da562b-kube-api-access-88vsf\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:39.006630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.006484 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6722f506-00e1-4596-aaab-34e7a8da562b-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:39.041004 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.038463 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:39.204946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.204868 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:39.333708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.333674 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:39.468569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.468455 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:39.486406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.486379 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" Nov 02 21:49:39.486595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.486406 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx" event={"ID":"6722f506-00e1-4596-aaab-34e7a8da562b","Type":"ContainerDied","Data":"2568d850519bcffb2eb74defbf43d6c0844017aaaca4f1c8ab7a03fc85701fe4"} Nov 02 21:49:39.486595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.486477 4441 scope.go:122] "RemoveContainer" containerID="18ee7b2c45e544d11ad421925f784af0cff36de74bf7dbde61d18c4be41c28ae" Nov 02 21:49:39.488754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.488724 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8" Nov 02 21:49:39.489136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.489049 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" Nov 02 21:49:39.489464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.489437 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj" event={"ID":"913f1206-00cc-4569-b38d-20897a9a2cd8","Type":"ContainerDied","Data":"1635a7e2b8fe8de54223f9048fa3c05a002316ac1bc5d7376cc064e3e457293f"} Nov 02 21:49:39.506126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.506099 4441 scope.go:122] "RemoveContainer" containerID="44c8ba81ed8c48da7c5d5c41c0e546740516da4e62ea2c8d9c8c66b57e4406ab" Nov 02 21:49:39.645730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.645701 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj"] Nov 02 21:49:39.692912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.692881 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-vhbmj"] Nov 02 21:49:39.766376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.766274 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8"] Nov 02 21:49:39.767634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.767611 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-jq4p8"] Nov 02 21:49:39.862921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.862895 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx"] Nov 02 21:49:39.868827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:39.868799 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-c2rhx"] Nov 02 21:49:40.272744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:40.272708 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:40.461603 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:40.461571 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:40.588371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:40.588335 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503d1f35-2825-4893-aacb-3070b61de20b" path="/var/lib/kubelet/pods/503d1f35-2825-4893-aacb-3070b61de20b/volumes" Nov 02 21:49:40.589049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:40.589024 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" path="/var/lib/kubelet/pods/6722f506-00e1-4596-aaab-34e7a8da562b/volumes" Nov 02 21:49:40.589683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:40.589663 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" path="/var/lib/kubelet/pods/913f1206-00cc-4569-b38d-20897a9a2cd8/volumes" Nov 02 21:49:40.617341 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:49:40.617311 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:49:42.384957 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.384897 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:42.445994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.445950 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl"] Nov 02 21:49:42.446592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446575 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" Nov 02 21:49:42.446641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446594 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ddcfe4-e1a2-4e98-adee-d30a1297583f" containerName="istio-proxy" Nov 02 21:49:42.446641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446625 4441 cpu_manager.go:414] "containerMap: removing container" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" Nov 02 21:49:42.446641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446630 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" Nov 02 21:49:42.446641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446641 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" Nov 02 21:49:42.446777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446646 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" Nov 02 21:49:42.446777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446661 4441 cpu_manager.go:414] "containerMap: removing container" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" Nov 02 21:49:42.446777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446666 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" Nov 02 21:49:42.446871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446802 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6722f506-00e1-4596-aaab-34e7a8da562b" containerName="istio-proxy" Nov 02 21:49:42.446871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446817 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="913f1206-00cc-4569-b38d-20897a9a2cd8" containerName="istio-proxy" Nov 02 21:49:42.446871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.446836 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="503d1f35-2825-4893-aacb-3070b61de20b" containerName="istio-proxy" Nov 02 21:49:42.452207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.452185 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.494347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.494314 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv"] Nov 02 21:49:42.498371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.498350 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.514134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.514111 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:42.556075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.554991 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl"] Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655151 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655228 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6604492-7b11-491c-87b9-dc752ba35206-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655257 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6604492-7b11-491c-87b9-dc752ba35206-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655282 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655301 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8wx\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-kube-api-access-jp8wx\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655327 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655344 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655769 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9aca-99d0-457b-b866-9e0ca409b0b3-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655811 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cp4k\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-kube-api-access-4cp4k\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655828 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655864 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655922 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655938 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.655982 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.656001 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.656024 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.656058 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.656083 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.660095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.657152 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv"] Nov 02 21:49:42.846992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.846782 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6604492-7b11-491c-87b9-dc752ba35206-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.846992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.846864 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6604492-7b11-491c-87b9-dc752ba35206-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.846992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.846916 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.846992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.846993 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8wx\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-kube-api-access-jp8wx\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847052 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847085 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847125 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9aca-99d0-457b-b866-9e0ca409b0b3-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847178 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cp4k\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-kube-api-access-4cp4k\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847210 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847293 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847338 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847369 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847427 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.847780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847464 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847496 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.847780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847541 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847596 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.847780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.847692 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.890997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.848328 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.899571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.899536 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.900729 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.900702 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.902023 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.901991 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9aca-99d0-457b-b866-9e0ca409b0b3-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.923319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.883875 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.930229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.930180 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.947029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.935142 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6604492-7b11-491c-87b9-dc752ba35206-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.947029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.935781 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.947029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.939031 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.947029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.939112 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.956649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.955881 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6604492-7b11-491c-87b9-dc752ba35206-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:42.975049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.967811 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.975049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.968911 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cp4k\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-kube-api-access-4cp4k\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.975049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.969435 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.975049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.970500 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.987011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.986939 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:42.995694 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:42.995667 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8wx\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-kube-api-access-jp8wx\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:43.021428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.021394 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:43.068127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.068099 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:43.112036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.112006 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:43.514488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.514404 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl"] Nov 02 21:49:43.519535 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.519514 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv"] Nov 02 21:49:43.543484 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:49:43.543457 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6604492_7b11_491c_87b9_dc752ba35206.slice/crio-cf88d61454c6807a4e043e239d70edbcd32e2abebac646a83d679bc960052f0d WatchSource:0}: Error finding container cf88d61454c6807a4e043e239d70edbcd32e2abebac646a83d679bc960052f0d: Status 404 returned error can't find the container with id cf88d61454c6807a4e043e239d70edbcd32e2abebac646a83d679bc960052f0d Nov 02 21:49:43.544360 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:49:43.544295 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74df9aca_99d0_457b_b866_9e0ca409b0b3.slice/crio-c9037edd9084a1311f879274ce8f1d3c1ca2393f18918a093229d253e3adbac4 WatchSource:0}: Error finding container c9037edd9084a1311f879274ce8f1d3c1ca2393f18918a093229d253e3adbac4: Status 404 returned error can't find the container with id c9037edd9084a1311f879274ce8f1d3c1ca2393f18918a093229d253e3adbac4 Nov 02 21:49:43.546613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.546579 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:43.546676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.546661 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:43.546746 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.546731 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:43.547210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.547186 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:43.547299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.547259 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:43.547299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.547291 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:49:43.557485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.557459 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" event={"ID":"74df9aca-99d0-457b-b866-9e0ca409b0b3","Type":"ContainerStarted","Data":"c9037edd9084a1311f879274ce8f1d3c1ca2393f18918a093229d253e3adbac4"} Nov 02 21:49:43.558805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.558783 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" event={"ID":"e6604492-7b11-491c-87b9-dc752ba35206","Type":"ContainerStarted","Data":"cf88d61454c6807a4e043e239d70edbcd32e2abebac646a83d679bc960052f0d"} Nov 02 21:49:43.961127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.961092 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:49:43.990865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.990832 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-socket\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:43.991079 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.990880 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-credential-socket\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:43.991204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.991176 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-socket" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:43.991276 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.991198 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-credential-socket" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:43.991409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.991382 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-envoy\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:43.991986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.991948 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:43.992090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.991992 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:43.993257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:43.993231 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-envoy" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:44.092905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.092820 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22456806-d042-419a-b631-c348f4775b44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22456806-d042-419a-b631-c348f4775b44-istiod-ca-cert\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:44.093576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.093555 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-istio-token\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-istio-token\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:44.094091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.094073 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-certs\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:44.094562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.094545 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22456806-d042-419a-b631-c348f4775b44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22456806-d042-419a-b631-c348f4775b44-istio-podinfo\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:44.094702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.094685 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-kube-api-access-lf8pj\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-kube-api-access-lf8pj\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:44.094985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.093467 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22456806-d042-419a-b631-c348f4775b44-istiod-ca-cert" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:44.095048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.094491 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-certs" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:44.095192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.095176 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-data\") pod \"22456806-d042-419a-b631-c348f4775b44\" (UID: \"22456806-d042-419a-b631-c348f4775b44\") " Nov 02 21:49:44.095809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.095792 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:44.095919 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.095908 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22456806-d042-419a-b631-c348f4775b44-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:44.096067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.096054 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:44.096329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.096314 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-data" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:44.100485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.100463 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22456806-d042-419a-b631-c348f4775b44-istio-podinfo" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:44.109011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.102142 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-kube-api-access-lf8pj" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "kube-api-access-lf8pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:44.109011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.105094 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-istio-token" pod "22456806-d042-419a-b631-c348f4775b44" (UID: "22456806-d042-419a-b631-c348f4775b44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:44.197154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.197117 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22456806-d042-419a-b631-c348f4775b44-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:44.197154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.197147 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:44.197154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.197156 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22456806-d042-419a-b631-c348f4775b44-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:44.197431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.197166 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf8pj\" (UniqueName: \"kubernetes.io/projected/22456806-d042-419a-b631-c348f4775b44-kube-api-access-lf8pj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:44.517048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.515594 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:44.568909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.568877 4441 generic.go:356] "Generic (PLEG): container finished" podID="22456806-d042-419a-b631-c348f4775b44" containerID="635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773" exitCode=0 Nov 02 21:49:44.569145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.568926 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" Nov 02 21:49:44.569145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.568959 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" event={"ID":"22456806-d042-419a-b631-c348f4775b44","Type":"ContainerDied","Data":"635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773"} Nov 02 21:49:44.569145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.569060 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9" event={"ID":"22456806-d042-419a-b631-c348f4775b44","Type":"ContainerDied","Data":"61b20a750654128f56559f549e91544b2dc96abfb986aa883864bb5138d97584"} Nov 02 21:49:44.569145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.569089 4441 scope.go:122] "RemoveContainer" containerID="635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773" Nov 02 21:49:44.643036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.643006 4441 scope.go:122] "RemoveContainer" containerID="ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d" Nov 02 21:49:44.645020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.644990 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" event={"ID":"e6604492-7b11-491c-87b9-dc752ba35206","Type":"ContainerStarted","Data":"0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6"} Nov 02 21:49:44.645159 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.645040 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" event={"ID":"74df9aca-99d0-457b-b866-9e0ca409b0b3","Type":"ContainerStarted","Data":"0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530"} Nov 02 21:49:44.656540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.656514 4441 scope.go:122] "RemoveContainer" containerID="635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773" Nov 02 21:49:44.656802 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:49:44.656783 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773\": container with ID starting with 635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773 not found: ID does not exist" containerID="635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773" Nov 02 21:49:44.656859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.656812 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773"} err="failed to get container status \"635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773\": rpc error: code = NotFound desc = could not find container \"635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773\": container with ID starting with 635b46ae328ab6c29b8128e562ea64e4943b2903735de8f86a23f3bdb1de3773 not found: ID does not exist" Nov 02 21:49:44.656859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.656829 4441 scope.go:122] "RemoveContainer" containerID="ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d" Nov 02 21:49:44.657060 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:49:44.657043 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d\": container with ID starting with ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d not found: ID does not exist" containerID="ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d" Nov 02 21:49:44.657115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.657066 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d"} err="failed to get container status \"ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d\": rpc error: code = NotFound desc = could not find container \"ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d\": container with ID starting with ff66360129a924143a1b76716cf8256c0e4fad2ee2ddae55e8da4597e764bd5d not found: ID does not exist" Nov 02 21:49:44.730993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.727670 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:44.754678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.754626 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podStartSLOduration=2.7546134159999998 podStartE2EDuration="2.754613416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:44.752745009 +0000 UTC m=+2214.308373340" watchObservedRunningTime="2026-11-02 21:49:44.754613416 +0000 UTC m=+2214.310241753" Nov 02 21:49:44.861455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.861412 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:44.886990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:44.886466 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podStartSLOduration=2.886441057 podStartE2EDuration="2.886441057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:44.864727091 +0000 UTC m=+2214.420355428" watchObservedRunningTime="2026-11-02 21:49:44.886441057 +0000 UTC m=+2214.442069390" Nov 02 21:49:45.015992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.013034 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9"] Nov 02 21:49:45.062366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.044684 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-66c72e-openshift-default-759cb84997-zklx9"] Nov 02 21:49:45.072998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.071523 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:45.072998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.071657 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 21:49:45.072998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.071699 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 21:49:45.134050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.133053 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:45.158213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.145101 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:49:45.158213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:45.145164 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:49:46.074002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:46.073301 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 21:49:46.074002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:46.073362 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 21:49:46.118993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:46.114307 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:49:46.118993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:46.114366 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:49:46.637358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:46.637312 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22456806-d042-419a-b631-c348f4775b44" path="/var/lib/kubelet/pods/22456806-d042-419a-b631-c348f4775b44/volumes" Nov 02 21:49:47.068584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:47.068542 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 21:49:47.068758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:47.068601 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 21:49:47.115991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:47.115262 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:49:47.115991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:47.115321 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:49:47.541041 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:47.539926 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:48.068589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:48.068542 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 21:49:48.068813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:48.068625 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 21:49:48.115001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:48.113708 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:49:48.115001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:48.113763 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:49:49.068427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:49.068387 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 21:49:49.068809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:49.068450 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 21:49:49.112991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:49.112339 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:49:49.112991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:49.112402 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:49:49.633195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:49.631176 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:49.819792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:49.819649 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:50.076991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:50.071335 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 21:49:50.076991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:50.071396 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 21:49:50.118993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:50.115330 4441 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:49:50.118993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:50.115389 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:49:51.073231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:51.073203 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:51.073708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:51.073686 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:51.074272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:51.074251 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:49:51.117036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:51.117016 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:51.117353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:51.117286 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:51.118025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:51.118003 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:49:52.545202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:52.545174 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:52.913207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:52.913177 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:53.556925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:53.556889 4441 scope.go:122] "RemoveContainer" containerID="6e8938ae4e9f076f2c7a154a39185009611ef7273e9975e40cacf182f6c29fb4" Nov 02 21:49:56.693604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:56.693564 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:58.044133 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.044096 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5"] Nov 02 21:49:58.091383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.091351 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0689b4f-slgl5"] Nov 02 21:49:58.317098 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.317061 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:58.548867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.548832 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:58.585402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.585323 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdd20b1-14eb-485c-8a7f-d1fd42660334" path="/var/lib/kubelet/pods/cbdd20b1-14eb-485c-8a7f-d1fd42660334/volumes" Nov 02 21:49:58.668716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.668573 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:58.792921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.792886 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv"] Nov 02 21:49:58.839651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:58.839572 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32de96-jltwv"] Nov 02 21:49:59.212552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:49:59.212465 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:00.587061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:00.587030 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4cb3598-70d6-404b-8255-545b453e1042" path="/var/lib/kubelet/pods/c4cb3598-70d6-404b-8255-545b453e1042/volumes" Nov 02 21:50:02.014796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.014759 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk"] Nov 02 21:50:02.015587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.015564 4441 cpu_manager.go:414] "containerMap: removing container" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" Nov 02 21:50:02.015711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.015588 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" Nov 02 21:50:02.015711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.015616 4441 cpu_manager.go:414] "containerMap: removing container" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" Nov 02 21:50:02.015711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.015624 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" Nov 02 21:50:02.015862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.015821 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" Nov 02 21:50:02.015862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.015842 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="22456806-d042-419a-b631-c348f4775b44" containerName="istio-proxy" Nov 02 21:50:02.021234 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.021212 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.073366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.073331 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk"] Nov 02 21:50:02.114090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.114061 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:02.114309 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.114072 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:50:02.114511 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.114498 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:02.114654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.114625 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:02.127056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.127016 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgcx\" (UniqueName: \"kubernetes.io/projected/c750bd06-fefd-4705-8136-b1804084625d-kube-api-access-9mgcx\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.127284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.127267 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c750bd06-fefd-4705-8136-b1804084625d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.127330 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.127307 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c750bd06-fefd-4705-8136-b1804084625d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.228463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.228429 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c750bd06-fefd-4705-8136-b1804084625d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.228463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.228474 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c750bd06-fefd-4705-8136-b1804084625d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.228742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.228539 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgcx\" (UniqueName: \"kubernetes.io/projected/c750bd06-fefd-4705-8136-b1804084625d-kube-api-access-9mgcx\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.229314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.229291 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c750bd06-fefd-4705-8136-b1804084625d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.230981 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.230946 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c750bd06-fefd-4705-8136-b1804084625d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.282396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.282319 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgcx\" (UniqueName: \"kubernetes.io/projected/c750bd06-fefd-4705-8136-b1804084625d-kube-api-access-9mgcx\") pod \"maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.334471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.334438 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:02.705992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.705087 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:02.715544 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.715501 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk"] Nov 02 21:50:02.719356 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:02.718767 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc750bd06_fefd_4705_8136_b1804084625d.slice/crio-32631f0ca31a05f522228e432733891a73e8129dd0d1ec1021723f14b1a9bc42 WatchSource:0}: Error finding container 32631f0ca31a05f522228e432733891a73e8129dd0d1ec1021723f14b1a9bc42: Status 404 returned error can't find the container with id 32631f0ca31a05f522228e432733891a73e8129dd0d1ec1021723f14b1a9bc42 Nov 02 21:50:02.903837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:02.903751 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" event={"ID":"c750bd06-fefd-4705-8136-b1804084625d","Type":"ContainerStarted","Data":"32631f0ca31a05f522228e432733891a73e8129dd0d1ec1021723f14b1a9bc42"} Nov 02 21:50:03.914334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:03.914238 4441 generic.go:356] "Generic (PLEG): container finished" podID="c750bd06-fefd-4705-8136-b1804084625d" containerID="e55887c3ee590be664e8e683f633a9e4ba83528ebf389353b21a10c0db71b6b7" exitCode=0 Nov 02 21:50:03.914770 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:03.914321 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" event={"ID":"c750bd06-fefd-4705-8136-b1804084625d","Type":"ContainerDied","Data":"e55887c3ee590be664e8e683f633a9e4ba83528ebf389353b21a10c0db71b6b7"} Nov 02 21:50:04.352532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:04.352499 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:04.612600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:04.612515 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:04.742833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:04.742792 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:04.843021 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:04.842989 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg"] Nov 02 21:50:04.846891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:04.846866 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1bf7d-2p9kg"] Nov 02 21:50:05.103065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.103033 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:05.136885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.136836 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:05.178432 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.178332 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c750bd06-fefd-4705-8136-b1804084625d-maas-api-openshift-ca" pod "c750bd06-fefd-4705-8136-b1804084625d" (UID: "c750bd06-fefd-4705-8136-b1804084625d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:05.178671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.178640 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c750bd06-fefd-4705-8136-b1804084625d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c750bd06-fefd-4705-8136-b1804084625d-maas-api-openshift-ca\") pod \"c750bd06-fefd-4705-8136-b1804084625d\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " Nov 02 21:50:05.178839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.178821 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c750bd06-fefd-4705-8136-b1804084625d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c750bd06-fefd-4705-8136-b1804084625d-maas-api-xks-ca\") pod \"c750bd06-fefd-4705-8136-b1804084625d\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " Nov 02 21:50:05.178949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.178934 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c750bd06-fefd-4705-8136-b1804084625d-kube-api-access-9mgcx\" (UniqueName: \"kubernetes.io/projected/c750bd06-fefd-4705-8136-b1804084625d-kube-api-access-9mgcx\") pod \"c750bd06-fefd-4705-8136-b1804084625d\" (UID: \"c750bd06-fefd-4705-8136-b1804084625d\") " Nov 02 21:50:05.179637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.179609 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c750bd06-fefd-4705-8136-b1804084625d-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:05.181273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.181245 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c750bd06-fefd-4705-8136-b1804084625d-kube-api-access-9mgcx" pod "c750bd06-fefd-4705-8136-b1804084625d" (UID: "c750bd06-fefd-4705-8136-b1804084625d"). InnerVolumeSpecName "kube-api-access-9mgcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:05.181512 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.181491 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c750bd06-fefd-4705-8136-b1804084625d-maas-api-xks-ca" pod "c750bd06-fefd-4705-8136-b1804084625d" (UID: "c750bd06-fefd-4705-8136-b1804084625d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:05.267356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.267320 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:05.281000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.280941 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mgcx\" (UniqueName: \"kubernetes.io/projected/c750bd06-fefd-4705-8136-b1804084625d-kube-api-access-9mgcx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:05.281000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.280998 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c750bd06-fefd-4705-8136-b1804084625d-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:05.482094 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.482016 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:05.931167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.931134 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" Nov 02 21:50:05.931167 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.931162 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk" event={"ID":"c750bd06-fefd-4705-8136-b1804084625d","Type":"ContainerDied","Data":"32631f0ca31a05f522228e432733891a73e8129dd0d1ec1021723f14b1a9bc42"} Nov 02 21:50:05.931380 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:05.931208 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32631f0ca31a05f522228e432733891a73e8129dd0d1ec1021723f14b1a9bc42" Nov 02 21:50:06.360651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:06.360591 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:06.585416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:06.585380 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3589ef44-a9a7-4b81-b130-ecf06b31e1ad" path="/var/lib/kubelet/pods/3589ef44-a9a7-4b81-b130-ecf06b31e1ad/volumes" Nov 02 21:50:06.934114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:06.932404 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:07.161003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:07.160460 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:09.529935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:09.529908 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:10.616319 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:50:10.616285 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:50:11.214795 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:11.214760 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:11.455690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:11.455646 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:11.594158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:11.594123 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:12.132536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.132503 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:12.691194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.691162 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd"] Nov 02 21:50:12.691764 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.691750 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c750bd06-fefd-4705-8136-b1804084625d" containerName="revoke-keys" Nov 02 21:50:12.691764 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.691765 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c750bd06-fefd-4705-8136-b1804084625d" containerName="revoke-keys" Nov 02 21:50:12.691884 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.691875 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c750bd06-fefd-4705-8136-b1804084625d" containerName="revoke-keys" Nov 02 21:50:12.695820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.695800 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.749659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.749629 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd"] Nov 02 21:50:12.791354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.791314 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:12.791354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.791334 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:12.791597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.791401 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:12.792226 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.792207 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:50:12.808508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.808480 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.808653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.808544 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.808733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.808706 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9mq\" (UniqueName: \"kubernetes.io/projected/d11190fa-b0e0-493d-b72a-be9883c6f515-kube-api-access-4s9mq\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.909998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.909946 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.910130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.910027 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.910180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.910137 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9mq\" (UniqueName: \"kubernetes.io/projected/d11190fa-b0e0-493d-b72a-be9883c6f515-kube-api-access-4s9mq\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.910631 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.910602 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.912586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.912551 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.963917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.963861 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9mq\" (UniqueName: \"kubernetes.io/projected/d11190fa-b0e0-493d-b72a-be9883c6f515-kube-api-access-4s9mq\") pod \"maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:12.977446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:12.977425 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x"] Nov 02 21:50:13.009434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:13.009413 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:13.085204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:13.084455 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4"] Nov 02 21:50:13.313156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:13.313118 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:13.447012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:13.443920 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:13.457086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:13.447518 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd"] Nov 02 21:50:14.168193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:14.168150 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" event={"ID":"d11190fa-b0e0-493d-b72a-be9883c6f515","Type":"ContainerStarted","Data":"8fb5aca7eea9dd8178b791f43ba1115312a0c97fdb177f45fd37cb260eedb070"} Nov 02 21:50:14.168510 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:14.168487 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" event={"ID":"d11190fa-b0e0-493d-b72a-be9883c6f515","Type":"ContainerStarted","Data":"52dde7bcc1b4390ec8166e9df8e7f3958569df87e54da5d6beed842d34a3c2d1"} Nov 02 21:50:14.278864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:14.278805 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" podStartSLOduration=2.278791259 podStartE2EDuration="2.278791259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:14.275931309 +0000 UTC m=+2243.831559648" watchObservedRunningTime="2026-11-02 21:50:14.278791259 +0000 UTC m=+2243.834419641" Nov 02 21:50:14.700939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:14.700843 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:15.178216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:15.178176 4441 generic.go:356] "Generic (PLEG): container finished" podID="d11190fa-b0e0-493d-b72a-be9883c6f515" containerID="8fb5aca7eea9dd8178b791f43ba1115312a0c97fdb177f45fd37cb260eedb070" exitCode=0 Nov 02 21:50:15.178415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:15.178224 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" event={"ID":"d11190fa-b0e0-493d-b72a-be9883c6f515","Type":"ContainerDied","Data":"8fb5aca7eea9dd8178b791f43ba1115312a0c97fdb177f45fd37cb260eedb070"} Nov 02 21:50:15.913116 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:15.913079 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:16.034779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.034743 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:16.348265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.348242 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:16.447376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.447303 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-openshift-ca\") pod \"d11190fa-b0e0-493d-b72a-be9883c6f515\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " Nov 02 21:50:16.447523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.447412 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-xks-ca\") pod \"d11190fa-b0e0-493d-b72a-be9883c6f515\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " Nov 02 21:50:16.447523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.447515 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11190fa-b0e0-493d-b72a-be9883c6f515-kube-api-access-4s9mq\" (UniqueName: \"kubernetes.io/projected/d11190fa-b0e0-493d-b72a-be9883c6f515-kube-api-access-4s9mq\") pod \"d11190fa-b0e0-493d-b72a-be9883c6f515\" (UID: \"d11190fa-b0e0-493d-b72a-be9883c6f515\") " Nov 02 21:50:16.447668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.447643 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-openshift-ca" pod "d11190fa-b0e0-493d-b72a-be9883c6f515" (UID: "d11190fa-b0e0-493d-b72a-be9883c6f515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:16.448035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.448009 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:16.449733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.449710 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-xks-ca" pod "d11190fa-b0e0-493d-b72a-be9883c6f515" (UID: "d11190fa-b0e0-493d-b72a-be9883c6f515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:16.449828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.449755 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11190fa-b0e0-493d-b72a-be9883c6f515-kube-api-access-4s9mq" pod "d11190fa-b0e0-493d-b72a-be9883c6f515" (UID: "d11190fa-b0e0-493d-b72a-be9883c6f515"). InnerVolumeSpecName "kube-api-access-4s9mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:16.549434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.549396 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11190fa-b0e0-493d-b72a-be9883c6f515-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:16.549434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:16.549436 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9mq\" (UniqueName: \"kubernetes.io/projected/d11190fa-b0e0-493d-b72a-be9883c6f515-kube-api-access-4s9mq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:17.100109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.100074 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks"] Nov 02 21:50:17.149462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.149434 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm"] Nov 02 21:50:17.155858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.155817 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp"] Nov 02 21:50:17.197223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.197191 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" Nov 02 21:50:17.197397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.197231 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd" event={"ID":"d11190fa-b0e0-493d-b72a-be9883c6f515","Type":"ContainerDied","Data":"52dde7bcc1b4390ec8166e9df8e7f3958569df87e54da5d6beed842d34a3c2d1"} Nov 02 21:50:17.197397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.197277 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52dde7bcc1b4390ec8166e9df8e7f3958569df87e54da5d6beed842d34a3c2d1" Nov 02 21:50:17.254095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.254051 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:17.826736 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.826702 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:17.991989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:17.987771 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" podUID="d7426317-bd74-4321-b8f6-9b31558ad3b1" containerName="payload-processing" containerID="cri-o://f387cffb88f18a96bff96fcb6807cce1a3fbaeeb47831d3a9ea05a42c95c0155" gracePeriod=25 Nov 02 21:50:18.018606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.004673 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:18.085202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.085167 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" podUID="7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" containerName="payload-pre-processing" containerID="cri-o://1b1f1f7917994b611cbc4d3272ff9c14f72d5bb1915c02657d1e9d162075357d" gracePeriod=25 Nov 02 21:50:18.219933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.219844 4441 generic.go:356] "Generic (PLEG): container finished" podID="d7426317-bd74-4321-b8f6-9b31558ad3b1" containerID="f387cffb88f18a96bff96fcb6807cce1a3fbaeeb47831d3a9ea05a42c95c0155" exitCode=0 Nov 02 21:50:18.220300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.219952 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" event={"ID":"d7426317-bd74-4321-b8f6-9b31558ad3b1","Type":"ContainerDied","Data":"f387cffb88f18a96bff96fcb6807cce1a3fbaeeb47831d3a9ea05a42c95c0155"} Nov 02 21:50:18.222160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.222068 4441 generic.go:356] "Generic (PLEG): container finished" podID="7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" containerID="1b1f1f7917994b611cbc4d3272ff9c14f72d5bb1915c02657d1e9d162075357d" exitCode=0 Nov 02 21:50:18.222160 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.222115 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" event={"ID":"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd","Type":"ContainerDied","Data":"1b1f1f7917994b611cbc4d3272ff9c14f72d5bb1915c02657d1e9d162075357d"} Nov 02 21:50:18.638374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.638351 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:50:18.641657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.641639 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:50:18.687228 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.687196 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-plugins-config-volume\") pod \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " Nov 02 21:50:18.687467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.687327 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7426317-bd74-4321-b8f6-9b31558ad3b1-kube-api-access-nzp8v\" (UniqueName: \"kubernetes.io/projected/d7426317-bd74-4321-b8f6-9b31558ad3b1-kube-api-access-nzp8v\") pod \"d7426317-bd74-4321-b8f6-9b31558ad3b1\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " Nov 02 21:50:18.687467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.687401 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-kube-api-access-s8knd\" (UniqueName: \"kubernetes.io/projected/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-kube-api-access-s8knd\") pod \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\" (UID: \"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd\") " Nov 02 21:50:18.687590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.687519 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7426317-bd74-4321-b8f6-9b31558ad3b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7426317-bd74-4321-b8f6-9b31558ad3b1-plugins-config-volume\") pod \"d7426317-bd74-4321-b8f6-9b31558ad3b1\" (UID: \"d7426317-bd74-4321-b8f6-9b31558ad3b1\") " Nov 02 21:50:18.687651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.687618 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-plugins-config-volume" pod "7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" (UID: "7f7f6c6f-e22f-4319-9407-c5dea7ee55cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:18.688719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.688317 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7426317-bd74-4321-b8f6-9b31558ad3b1-plugins-config-volume" pod "d7426317-bd74-4321-b8f6-9b31558ad3b1" (UID: "d7426317-bd74-4321-b8f6-9b31558ad3b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:18.688719 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.688362 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:18.690174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.690148 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7426317-bd74-4321-b8f6-9b31558ad3b1-kube-api-access-nzp8v" pod "d7426317-bd74-4321-b8f6-9b31558ad3b1" (UID: "d7426317-bd74-4321-b8f6-9b31558ad3b1"). InnerVolumeSpecName "kube-api-access-nzp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:18.690350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.690331 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-kube-api-access-s8knd" pod "7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" (UID: "7f7f6c6f-e22f-4319-9407-c5dea7ee55cd"). InnerVolumeSpecName "kube-api-access-s8knd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:18.790440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.790394 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzp8v\" (UniqueName: \"kubernetes.io/projected/d7426317-bd74-4321-b8f6-9b31558ad3b1-kube-api-access-nzp8v\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:18.790440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.790439 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8knd\" (UniqueName: \"kubernetes.io/projected/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd-kube-api-access-s8knd\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:18.790708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:18.790454 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7426317-bd74-4321-b8f6-9b31558ad3b1-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:19.232486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.232452 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" Nov 02 21:50:19.234007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.233957 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x" event={"ID":"d7426317-bd74-4321-b8f6-9b31558ad3b1","Type":"ContainerDied","Data":"455362f2a3e5cc5782c0e54874a30e0a57764c0412a4d0b95a7c70d6fd21302d"} Nov 02 21:50:19.234170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.234047 4441 scope.go:122] "RemoveContainer" containerID="f387cffb88f18a96bff96fcb6807cce1a3fbaeeb47831d3a9ea05a42c95c0155" Nov 02 21:50:19.238851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.238829 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" Nov 02 21:50:19.239235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.239210 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4" event={"ID":"7f7f6c6f-e22f-4319-9407-c5dea7ee55cd","Type":"ContainerDied","Data":"2880cbaa1239c03993e2292fa6d2b04027efd10fd33261bf7213d61e0bf65e97"} Nov 02 21:50:19.248557 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.248537 4441 scope.go:122] "RemoveContainer" containerID="1b1f1f7917994b611cbc4d3272ff9c14f72d5bb1915c02657d1e9d162075357d" Nov 02 21:50:19.392722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.392639 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x"] Nov 02 21:50:19.397310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.397277 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-t7x6x"] Nov 02 21:50:19.497762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.497717 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4"] Nov 02 21:50:19.501914 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.501885 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-hvlc4"] Nov 02 21:50:19.637998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:19.637945 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:20.588250 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:20.588214 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" path="/var/lib/kubelet/pods/7f7f6c6f-e22f-4319-9407-c5dea7ee55cd/volumes" Nov 02 21:50:20.588672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:20.588656 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7426317-bd74-4321-b8f6-9b31558ad3b1" path="/var/lib/kubelet/pods/d7426317-bd74-4321-b8f6-9b31558ad3b1/volumes" Nov 02 21:50:23.434670 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:23.434634 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s"] Nov 02 21:50:23.667958 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:23.667927 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:23.801961 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:23.801860 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:24.655927 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:24.655871 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:25.149548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:25.149510 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:25.375407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:25.375373 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:28.435313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.435264 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" podUID="0f385b13-7059-40f1-876f-d9f0a0eaf369" containerName="payload-pre-processing" containerID="cri-o://9ef8454f649a9e2b517d0f086edf76e3757f2e96b99d607bac9eb640031ced14" gracePeriod=25 Nov 02 21:50:28.984881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.984843 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552"] Nov 02 21:50:28.985564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985541 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d11190fa-b0e0-493d-b72a-be9883c6f515" containerName="revoke-keys" Nov 02 21:50:28.985696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985565 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11190fa-b0e0-493d-b72a-be9883c6f515" containerName="revoke-keys" Nov 02 21:50:28.985696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985585 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d7426317-bd74-4321-b8f6-9b31558ad3b1" containerName="payload-processing" Nov 02 21:50:28.985696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985593 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7426317-bd74-4321-b8f6-9b31558ad3b1" containerName="payload-processing" Nov 02 21:50:28.985696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985607 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" containerName="payload-pre-processing" Nov 02 21:50:28.985696 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985615 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" containerName="payload-pre-processing" Nov 02 21:50:28.985959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985769 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11190fa-b0e0-493d-b72a-be9883c6f515" containerName="revoke-keys" Nov 02 21:50:28.985959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985787 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7426317-bd74-4321-b8f6-9b31558ad3b1" containerName="payload-processing" Nov 02 21:50:28.985959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.985800 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7f6c6f-e22f-4319-9407-c5dea7ee55cd" containerName="payload-pre-processing" Nov 02 21:50:28.990811 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:28.990789 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.040673 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.040642 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk"] Nov 02 21:50:29.045499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.045466 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.046601 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.046571 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552"] Nov 02 21:50:29.054584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.053874 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.054584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.054106 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wmf\" (UniqueName: \"kubernetes.io/projected/c48e7c50-073c-4d01-a89e-31b107577c1e-kube-api-access-z5wmf\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.054584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.054367 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.082640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.082613 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:29.082799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.082781 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:29.082881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.082797 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:50:29.085418 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.085402 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:29.102463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.102436 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk"] Nov 02 21:50:29.155161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.155130 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.155299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.155275 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.155423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.155403 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.155515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.155445 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.155630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.155603 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjh4\" (UniqueName: \"kubernetes.io/projected/3789c0de-a398-48b6-ac6e-09b053059650-kube-api-access-nzjh4\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.155763 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.155643 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wmf\" (UniqueName: \"kubernetes.io/projected/c48e7c50-073c-4d01-a89e-31b107577c1e-kube-api-access-z5wmf\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.156269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.156249 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.157632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.157606 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.209791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.209753 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wmf\" (UniqueName: \"kubernetes.io/projected/c48e7c50-073c-4d01-a89e-31b107577c1e-kube-api-access-z5wmf\") pod \"maas-api-revoke-sub-8c2fc43d-65552\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.256565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.256489 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.256745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.256612 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjh4\" (UniqueName: \"kubernetes.io/projected/3789c0de-a398-48b6-ac6e-09b053059650-kube-api-access-nzjh4\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.256745 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.256679 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.257571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.257539 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.259266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.259242 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.306602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.306573 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:29.310356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.310328 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjh4\" (UniqueName: \"kubernetes.io/projected/3789c0de-a398-48b6-ac6e-09b053059650-kube-api-access-nzjh4\") pod \"maas-api-revoke-sub-146703ee-scclk\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.367031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.364872 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:29.701790 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:29.701741 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48e7c50_073c_4d01_a89e_31b107577c1e.slice/crio-a56e68d080d5f316bdb80f015af77010f5ca4ea483e3b908f5caefd87e13939b WatchSource:0}: Error finding container a56e68d080d5f316bdb80f015af77010f5ca4ea483e3b908f5caefd87e13939b: Status 404 returned error can't find the container with id a56e68d080d5f316bdb80f015af77010f5ca4ea483e3b908f5caefd87e13939b Nov 02 21:50:29.702525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.702501 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552"] Nov 02 21:50:29.803033 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:29.803007 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3789c0de_a398_48b6_ac6e_09b053059650.slice/crio-9aed05802eb9cb1b056c4aafb5628ef2f3a3f40e6388ec43145403142ef9d033 WatchSource:0}: Error finding container 9aed05802eb9cb1b056c4aafb5628ef2f3a3f40e6388ec43145403142ef9d033: Status 404 returned error can't find the container with id 9aed05802eb9cb1b056c4aafb5628ef2f3a3f40e6388ec43145403142ef9d033 Nov 02 21:50:29.803885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.803864 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk"] Nov 02 21:50:29.947413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:29.947386 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:30.345724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:30.345687 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" event={"ID":"3789c0de-a398-48b6-ac6e-09b053059650","Type":"ContainerStarted","Data":"8d2cad65e8742d51dbf841dbb322ce848700efa3f3112e9042ff1a179949a9f7"} Nov 02 21:50:30.345898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:30.345751 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" event={"ID":"3789c0de-a398-48b6-ac6e-09b053059650","Type":"ContainerStarted","Data":"9aed05802eb9cb1b056c4aafb5628ef2f3a3f40e6388ec43145403142ef9d033"} Nov 02 21:50:30.347515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:30.347485 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" event={"ID":"c48e7c50-073c-4d01-a89e-31b107577c1e","Type":"ContainerStarted","Data":"cfb7dddf6dc70cd1287748df24038af1114f35ae1e37277dbe1c0a86c22d0a8e"} Nov 02 21:50:30.347647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:30.347541 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" event={"ID":"c48e7c50-073c-4d01-a89e-31b107577c1e","Type":"ContainerStarted","Data":"a56e68d080d5f316bdb80f015af77010f5ca4ea483e3b908f5caefd87e13939b"} Nov 02 21:50:30.456602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:30.456539 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" podStartSLOduration=2.456525866 podStartE2EDuration="2.456525866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:30.453072265 +0000 UTC m=+2260.008700616" watchObservedRunningTime="2026-11-02 21:50:30.456525866 +0000 UTC m=+2260.012154202" Nov 02 21:50:30.564006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:30.563931 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" podStartSLOduration=2.563918223 podStartE2EDuration="2.563918223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:30.56049837 +0000 UTC m=+2260.116126693" watchObservedRunningTime="2026-11-02 21:50:30.563918223 +0000 UTC m=+2260.119546560" Nov 02 21:50:31.357255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:31.357219 4441 generic.go:356] "Generic (PLEG): container finished" podID="3789c0de-a398-48b6-ac6e-09b053059650" containerID="8d2cad65e8742d51dbf841dbb322ce848700efa3f3112e9042ff1a179949a9f7" exitCode=0 Nov 02 21:50:31.357728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:31.357301 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" event={"ID":"3789c0de-a398-48b6-ac6e-09b053059650","Type":"ContainerDied","Data":"8d2cad65e8742d51dbf841dbb322ce848700efa3f3112e9042ff1a179949a9f7"} Nov 02 21:50:31.359289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:31.359260 4441 generic.go:356] "Generic (PLEG): container finished" podID="c48e7c50-073c-4d01-a89e-31b107577c1e" containerID="cfb7dddf6dc70cd1287748df24038af1114f35ae1e37277dbe1c0a86c22d0a8e" exitCode=0 Nov 02 21:50:31.359437 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:31.359340 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" event={"ID":"c48e7c50-073c-4d01-a89e-31b107577c1e","Type":"ContainerDied","Data":"cfb7dddf6dc70cd1287748df24038af1114f35ae1e37277dbe1c0a86c22d0a8e"} Nov 02 21:50:32.122858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.122817 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" containerID="cri-o://55b6eaf9a80c880057051c92b1b27e535bdd4229c7ba2383a16a931562fd3aee" gracePeriod=45 Nov 02 21:50:32.170225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.170190 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" containerID="cri-o://646ad6d52da12e1def53b5ef30e82d13c415d1aeded8b81990c5c31e1a1740e3" gracePeriod=45 Nov 02 21:50:32.175805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.175776 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" containerID="cri-o://0b0f7c4cd37393a5cf5202b96d6e4ec9537b51fb057a4be3f0ed34b77d071eee" gracePeriod=45 Nov 02 21:50:32.382865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.382505 4441 generic.go:356] "Generic (PLEG): container finished" podID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerID="55b6eaf9a80c880057051c92b1b27e535bdd4229c7ba2383a16a931562fd3aee" exitCode=0 Nov 02 21:50:32.382865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.382749 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" event={"ID":"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f","Type":"ContainerDied","Data":"55b6eaf9a80c880057051c92b1b27e535bdd4229c7ba2383a16a931562fd3aee"} Nov 02 21:50:32.386073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.386036 4441 generic.go:356] "Generic (PLEG): container finished" podID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerID="0b0f7c4cd37393a5cf5202b96d6e4ec9537b51fb057a4be3f0ed34b77d071eee" exitCode=0 Nov 02 21:50:32.386210 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.386104 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" event={"ID":"17df70f7-9cbe-452b-96ff-0edad519ec94","Type":"ContainerDied","Data":"0b0f7c4cd37393a5cf5202b96d6e4ec9537b51fb057a4be3f0ed34b77d071eee"} Nov 02 21:50:32.387895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.387873 4441 generic.go:356] "Generic (PLEG): container finished" podID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerID="646ad6d52da12e1def53b5ef30e82d13c415d1aeded8b81990c5c31e1a1740e3" exitCode=0 Nov 02 21:50:32.388030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.387945 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" event={"ID":"8f454f72-4073-4bbe-9f10-c11ba9df402f","Type":"ContainerDied","Data":"646ad6d52da12e1def53b5ef30e82d13c415d1aeded8b81990c5c31e1a1740e3"} Nov 02 21:50:32.422275 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.411429 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused" Nov 02 21:50:32.504742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.504172 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:50:32.537198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.536171 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:50:32.544154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.543244 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:50:32.621284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.621173 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-dshm\") pod \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " Nov 02 21:50:32.621284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.621236 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tls-certs\") pod \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " Nov 02 21:50:32.621532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.621334 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-kube-api-access-gqmnw\" (UniqueName: \"kubernetes.io/projected/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-kube-api-access-gqmnw\") pod \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " Nov 02 21:50:32.621532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.621365 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-model-cache\") pod \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " Nov 02 21:50:32.621532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.621411 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tmp-dir\") pod \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " Nov 02 21:50:32.621702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.621536 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-home\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-home\") pod \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\" (UID: \"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f\") " Nov 02 21:50:32.623843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.623285 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-home" pod "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" (UID: "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.623843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.623560 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-model-cache" pod "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" (UID: "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.624130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.624047 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tmp-dir" pod "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" (UID: "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.628097 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.628071 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-kube-api-access-gqmnw" pod "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" (UID: "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f"). InnerVolumeSpecName "kube-api-access-gqmnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:32.629161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.629139 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-dshm" pod "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" (UID: "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.630035 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.630000 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tls-certs" pod "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" (UID: "b2afc8e7-362d-4ece-9c7a-ab9bd02af26f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:32.639282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.639258 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:32.642217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.642188 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:32.723580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.723455 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-model-cache\") pod \"8f454f72-4073-4bbe-9f10-c11ba9df402f\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " Nov 02 21:50:32.723875 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.723843 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-tmp-dir\") pod \"17df70f7-9cbe-452b-96ff-0edad519ec94\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " Nov 02 21:50:32.724091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.723910 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-model-cache\") pod \"17df70f7-9cbe-452b-96ff-0edad519ec94\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.725866 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17df70f7-9cbe-452b-96ff-0edad519ec94-tls-certs\" (UniqueName: \"kubernetes.io/secret/17df70f7-9cbe-452b-96ff-0edad519ec94-tls-certs\") pod \"17df70f7-9cbe-452b-96ff-0edad519ec94\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.725998 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-dshm\") pod \"8f454f72-4073-4bbe-9f10-c11ba9df402f\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.726032 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17df70f7-9cbe-452b-96ff-0edad519ec94-kube-api-access-kmvql\" (UniqueName: \"kubernetes.io/projected/17df70f7-9cbe-452b-96ff-0edad519ec94-kube-api-access-kmvql\") pod \"17df70f7-9cbe-452b-96ff-0edad519ec94\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.726083 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f454f72-4073-4bbe-9f10-c11ba9df402f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f454f72-4073-4bbe-9f10-c11ba9df402f-tls-certs\") pod \"8f454f72-4073-4bbe-9f10-c11ba9df402f\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.726160 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-tmp-dir\") pod \"8f454f72-4073-4bbe-9f10-c11ba9df402f\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.726208 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-home\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-home\") pod \"17df70f7-9cbe-452b-96ff-0edad519ec94\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.726237 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-home\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-home\") pod \"8f454f72-4073-4bbe-9f10-c11ba9df402f\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.726270 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-dshm\") pod \"17df70f7-9cbe-452b-96ff-0edad519ec94\" (UID: \"17df70f7-9cbe-452b-96ff-0edad519ec94\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.726300 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f454f72-4073-4bbe-9f10-c11ba9df402f-kube-api-access-w5knx\" (UniqueName: \"kubernetes.io/projected/8f454f72-4073-4bbe-9f10-c11ba9df402f-kube-api-access-w5knx\") pod \"8f454f72-4073-4bbe-9f10-c11ba9df402f\" (UID: \"8f454f72-4073-4bbe-9f10-c11ba9df402f\") " Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.727517 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-home" pod "17df70f7-9cbe-452b-96ff-0edad519ec94" (UID: "17df70f7-9cbe-452b-96ff-0edad519ec94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.727713 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-home" pod "8f454f72-4073-4bbe-9f10-c11ba9df402f" (UID: "8f454f72-4073-4bbe-9f10-c11ba9df402f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.728269 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-model-cache" pod "17df70f7-9cbe-452b-96ff-0edad519ec94" (UID: "17df70f7-9cbe-452b-96ff-0edad519ec94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.728765 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.728447 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-model-cache" pod "8f454f72-4073-4bbe-9f10-c11ba9df402f" (UID: "8f454f72-4073-4bbe-9f10-c11ba9df402f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729049 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-tmp-dir" pod "17df70f7-9cbe-452b-96ff-0edad519ec94" (UID: "17df70f7-9cbe-452b-96ff-0edad519ec94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729400 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729417 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729427 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729439 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729452 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729471 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729480 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729489 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729499 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmnw\" (UniqueName: \"kubernetes.io/projected/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-kube-api-access-gqmnw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729508 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.730548 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.729519 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.732796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.730692 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-tmp-dir" pod "8f454f72-4073-4bbe-9f10-c11ba9df402f" (UID: "8f454f72-4073-4bbe-9f10-c11ba9df402f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.733539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.733387 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f454f72-4073-4bbe-9f10-c11ba9df402f-tls-certs" pod "8f454f72-4073-4bbe-9f10-c11ba9df402f" (UID: "8f454f72-4073-4bbe-9f10-c11ba9df402f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:32.733539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.733501 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-dshm" pod "17df70f7-9cbe-452b-96ff-0edad519ec94" (UID: "17df70f7-9cbe-452b-96ff-0edad519ec94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.733708 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.733558 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17df70f7-9cbe-452b-96ff-0edad519ec94-tls-certs" pod "17df70f7-9cbe-452b-96ff-0edad519ec94" (UID: "17df70f7-9cbe-452b-96ff-0edad519ec94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:32.733899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.733874 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-dshm" pod "8f454f72-4073-4bbe-9f10-c11ba9df402f" (UID: "8f454f72-4073-4bbe-9f10-c11ba9df402f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:32.734204 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.734172 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17df70f7-9cbe-452b-96ff-0edad519ec94-kube-api-access-kmvql" pod "17df70f7-9cbe-452b-96ff-0edad519ec94" (UID: "17df70f7-9cbe-452b-96ff-0edad519ec94"). InnerVolumeSpecName "kube-api-access-kmvql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:32.734419 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.734394 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f454f72-4073-4bbe-9f10-c11ba9df402f-kube-api-access-w5knx" pod "8f454f72-4073-4bbe-9f10-c11ba9df402f" (UID: "8f454f72-4073-4bbe-9f10-c11ba9df402f"). InnerVolumeSpecName "kube-api-access-w5knx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:32.830174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830144 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48e7c50-073c-4d01-a89e-31b107577c1e-kube-api-access-z5wmf\" (UniqueName: \"kubernetes.io/projected/c48e7c50-073c-4d01-a89e-31b107577c1e-kube-api-access-z5wmf\") pod \"c48e7c50-073c-4d01-a89e-31b107577c1e\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " Nov 02 21:50:32.830355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830212 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-openshift-ca\") pod \"3789c0de-a398-48b6-ac6e-09b053059650\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " Nov 02 21:50:32.830436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830380 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-xks-ca\") pod \"3789c0de-a398-48b6-ac6e-09b053059650\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " Nov 02 21:50:32.830436 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830424 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-xks-ca\") pod \"c48e7c50-073c-4d01-a89e-31b107577c1e\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " Nov 02 21:50:32.830552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830487 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-openshift-ca\") pod \"c48e7c50-073c-4d01-a89e-31b107577c1e\" (UID: \"c48e7c50-073c-4d01-a89e-31b107577c1e\") " Nov 02 21:50:32.830552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830525 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3789c0de-a398-48b6-ac6e-09b053059650-kube-api-access-nzjh4\" (UniqueName: \"kubernetes.io/projected/3789c0de-a398-48b6-ac6e-09b053059650-kube-api-access-nzjh4\") pod \"3789c0de-a398-48b6-ac6e-09b053059650\" (UID: \"3789c0de-a398-48b6-ac6e-09b053059650\") " Nov 02 21:50:32.830671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830618 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-openshift-ca" pod "3789c0de-a398-48b6-ac6e-09b053059650" (UID: "3789c0de-a398-48b6-ac6e-09b053059650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:32.830894 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.830863 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-openshift-ca" pod "c48e7c50-073c-4d01-a89e-31b107577c1e" (UID: "c48e7c50-073c-4d01-a89e-31b107577c1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:32.831224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831206 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831229 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17df70f7-9cbe-452b-96ff-0edad519ec94-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831243 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5knx\" (UniqueName: \"kubernetes.io/projected/8f454f72-4073-4bbe-9f10-c11ba9df402f-kube-api-access-w5knx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831257 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831270 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831284 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17df70f7-9cbe-452b-96ff-0edad519ec94-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831297 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f454f72-4073-4bbe-9f10-c11ba9df402f-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831309 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvql\" (UniqueName: \"kubernetes.io/projected/17df70f7-9cbe-452b-96ff-0edad519ec94-kube-api-access-kmvql\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.831329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.831324 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f454f72-4073-4bbe-9f10-c11ba9df402f-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.832586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.832529 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-xks-ca" pod "3789c0de-a398-48b6-ac6e-09b053059650" (UID: "3789c0de-a398-48b6-ac6e-09b053059650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:32.832747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.832728 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-xks-ca" pod "c48e7c50-073c-4d01-a89e-31b107577c1e" (UID: "c48e7c50-073c-4d01-a89e-31b107577c1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:32.832858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.832821 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48e7c50-073c-4d01-a89e-31b107577c1e-kube-api-access-z5wmf" pod "c48e7c50-073c-4d01-a89e-31b107577c1e" (UID: "c48e7c50-073c-4d01-a89e-31b107577c1e"). InnerVolumeSpecName "kube-api-access-z5wmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:32.832998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.832919 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3789c0de-a398-48b6-ac6e-09b053059650-kube-api-access-nzjh4" pod "3789c0de-a398-48b6-ac6e-09b053059650" (UID: "3789c0de-a398-48b6-ac6e-09b053059650"). InnerVolumeSpecName "kube-api-access-nzjh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:32.932716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.932630 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3789c0de-a398-48b6-ac6e-09b053059650-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.932716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.932661 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48e7c50-073c-4d01-a89e-31b107577c1e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.932716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.932675 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjh4\" (UniqueName: \"kubernetes.io/projected/3789c0de-a398-48b6-ac6e-09b053059650-kube-api-access-nzjh4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.932716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:32.932687 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wmf\" (UniqueName: \"kubernetes.io/projected/c48e7c50-073c-4d01-a89e-31b107577c1e-kube-api-access-z5wmf\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:33.398916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.398890 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" Nov 02 21:50:33.399413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.398931 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552" event={"ID":"c48e7c50-073c-4d01-a89e-31b107577c1e","Type":"ContainerDied","Data":"a56e68d080d5f316bdb80f015af77010f5ca4ea483e3b908f5caefd87e13939b"} Nov 02 21:50:33.399413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.398995 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a56e68d080d5f316bdb80f015af77010f5ca4ea483e3b908f5caefd87e13939b" Nov 02 21:50:33.401032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.401012 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" Nov 02 21:50:33.401191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.401034 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp" event={"ID":"17df70f7-9cbe-452b-96ff-0edad519ec94","Type":"ContainerDied","Data":"39843b96f8dc2f3dc0b6ca608f9ca2e661c446c58557971a26fb044082b25c4d"} Nov 02 21:50:33.401191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.401089 4441 scope.go:122] "RemoveContainer" containerID="0b0f7c4cd37393a5cf5202b96d6e4ec9537b51fb057a4be3f0ed34b77d071eee" Nov 02 21:50:33.403311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.403282 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" event={"ID":"8f454f72-4073-4bbe-9f10-c11ba9df402f","Type":"ContainerDied","Data":"341178be947056c4871eb28ea286bbebf5cadcb8eab1b44f8d81c5514c4a9686"} Nov 02 21:50:33.403449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.403375 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" Nov 02 21:50:33.410723 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.410698 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" event={"ID":"3789c0de-a398-48b6-ac6e-09b053059650","Type":"ContainerDied","Data":"9aed05802eb9cb1b056c4aafb5628ef2f3a3f40e6388ec43145403142ef9d033"} Nov 02 21:50:33.410829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.410742 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aed05802eb9cb1b056c4aafb5628ef2f3a3f40e6388ec43145403142ef9d033" Nov 02 21:50:33.410829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.410802 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk" Nov 02 21:50:33.412864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.412842 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" event={"ID":"b2afc8e7-362d-4ece-9c7a-ab9bd02af26f","Type":"ContainerDied","Data":"85b169daf970ac8e4728d270cb0bcfcd62ad29d4f031917d21dc33363f743541"} Nov 02 21:50:33.413149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.412879 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks" Nov 02 21:50:33.416669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.416475 4441 scope.go:122] "RemoveContainer" containerID="646ad6d52da12e1def53b5ef30e82d13c415d1aeded8b81990c5c31e1a1740e3" Nov 02 21:50:33.437943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.437915 4441 scope.go:122] "RemoveContainer" containerID="55b6eaf9a80c880057051c92b1b27e535bdd4229c7ba2383a16a931562fd3aee" Nov 02 21:50:33.466924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.466893 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:50:33.610181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.610146 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp"] Nov 02 21:50:33.615289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.615260 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/simulator-w0-dd6574-kserve-cf6ccf85c-lw4wp"] Nov 02 21:50:33.719171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.719083 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm"] Nov 02 21:50:33.723010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.722985 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/premium-w0-dd6574-kserve-68794fdc76-5jqpm"] Nov 02 21:50:33.873753 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.873707 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks"] Nov 02 21:50:33.877900 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:33.877874 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-dd6574/distinct-2-w0-dd6574-kserve-7896b88cf6-79xks"] Nov 02 21:50:34.589893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:34.589847 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" path="/var/lib/kubelet/pods/17df70f7-9cbe-452b-96ff-0edad519ec94/volumes" Nov 02 21:50:34.590558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:34.590536 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" path="/var/lib/kubelet/pods/8f454f72-4073-4bbe-9f10-c11ba9df402f/volumes" Nov 02 21:50:34.591152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:34.591134 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" path="/var/lib/kubelet/pods/b2afc8e7-362d-4ece-9c7a-ab9bd02af26f/volumes" Nov 02 21:50:34.633451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:34.633415 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:34.800999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:34.799708 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:34.933807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:34.930336 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:36.549698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:36.549662 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" podUID="0f385b13-7059-40f1-876f-d9f0a0eaf369" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:50:38.790991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:38.778619 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:39.211546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.211507 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:39.693993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.687102 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:39.990051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.989442 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn"] Nov 02 21:50:39.990514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990426 4441 cpu_manager.go:414] "containerMap: removing container" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" Nov 02 21:50:39.990514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990448 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" Nov 02 21:50:39.990514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990467 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" Nov 02 21:50:39.990514 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990476 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990520 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990528 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990541 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3789c0de-a398-48b6-ac6e-09b053059650" containerName="revoke-keys" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990549 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3789c0de-a398-48b6-ac6e-09b053059650" containerName="revoke-keys" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990563 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c48e7c50-073c-4d01-a89e-31b107577c1e" containerName="revoke-keys" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990570 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48e7c50-073c-4d01-a89e-31b107577c1e" containerName="revoke-keys" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990702 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f454f72-4073-4bbe-9f10-c11ba9df402f" containerName="main" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990722 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2afc8e7-362d-4ece-9c7a-ab9bd02af26f" containerName="main" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990736 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3789c0de-a398-48b6-ac6e-09b053059650" containerName="revoke-keys" Nov 02 21:50:39.990741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990745 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48e7c50-073c-4d01-a89e-31b107577c1e" containerName="revoke-keys" Nov 02 21:50:39.991284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:39.990760 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="17df70f7-9cbe-452b-96ff-0edad519ec94" containerName="main" Nov 02 21:50:40.008267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.008111 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.059137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.059108 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d4f799-openshift-default-dockercfg-twq6l\"" Nov 02 21:50:40.094779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.094278 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:40.107012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.106940 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-credential-socket\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107008 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2719ffe-8af4-4118-bf17-be831cfb32da-istiod-ca-cert\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107100 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-envoy\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107193 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-certs\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107327 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-data\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107389 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj5tt\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-kube-api-access-fj5tt\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107427 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-podinfo\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107456 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-token\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.107562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.107493 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-socket\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.113916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.113887 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn"] Nov 02 21:50:40.208486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208447 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-envoy\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.208666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208563 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-certs\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.208750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208662 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-data\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.208750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208708 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj5tt\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-kube-api-access-fj5tt\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.208750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208744 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-podinfo\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.208898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208773 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-token\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.208898 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208807 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-socket\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.209053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208905 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-credential-socket\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.209053 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.208936 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2719ffe-8af4-4118-bf17-be831cfb32da-istiod-ca-cert\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.209500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.209472 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-data\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.209627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.209501 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-socket\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.209627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.209564 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-credential-socket\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.209760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.209737 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2719ffe-8af4-4118-bf17-be831cfb32da-istiod-ca-cert\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.209816 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.209759 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-certs\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.211605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.211581 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-envoy\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.211706 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.211626 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-podinfo\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.229491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.229069 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:40.279472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.275547 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj5tt\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-kube-api-access-fj5tt\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.280414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.280383 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-token\") pod \"e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.326685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.326650 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:40.623014 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:50:40.620329 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:50:40.776312 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:40.776276 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2719ffe_8af4_4118_bf17_be831cfb32da.slice/crio-ebf3fc4fe82ec486791a7edc6ed57eb2f8473da67dcf7d3fc2e99d5da6f3564c WatchSource:0}: Error finding container ebf3fc4fe82ec486791a7edc6ed57eb2f8473da67dcf7d3fc2e99d5da6f3564c: Status 404 returned error can't find the container with id ebf3fc4fe82ec486791a7edc6ed57eb2f8473da67dcf7d3fc2e99d5da6f3564c Nov 02 21:50:40.779829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.779800 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn"] Nov 02 21:50:40.780338 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.780302 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:50:40.780452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.780385 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:50:40.780452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:40.780431 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:50:41.525645 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:41.525603 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" event={"ID":"f2719ffe-8af4-4118-bf17-be831cfb32da","Type":"ContainerStarted","Data":"7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16"} Nov 02 21:50:41.526192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:41.525675 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" event={"ID":"f2719ffe-8af4-4118-bf17-be831cfb32da","Type":"ContainerStarted","Data":"ebf3fc4fe82ec486791a7edc6ed57eb2f8473da67dcf7d3fc2e99d5da6f3564c"} Nov 02 21:50:41.637262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:41.637207 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podStartSLOduration=2.637186921 podStartE2EDuration="2.637186921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:41.635057042 +0000 UTC m=+2271.190685405" watchObservedRunningTime="2026-11-02 21:50:41.637186921 +0000 UTC m=+2271.192815318" Nov 02 21:50:42.327407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:42.327371 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:42.334871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:42.334842 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:42.533303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:42.533272 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:42.534598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:42.534565 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:50:44.358001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.357957 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:44.550812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.550762 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:44.551223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.551198 4441 generic.go:356] "Generic (PLEG): container finished" podID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerID="921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549" exitCode=0 Nov 02 21:50:44.551329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.551280 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" event={"ID":"03da32a5-06a1-46e4-8379-154d9524ad4d","Type":"ContainerDied","Data":"921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549"} Nov 02 21:50:44.551822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.551801 4441 scope.go:122] "RemoveContainer" containerID="921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549" Nov 02 21:50:44.553422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.553394 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:50:44.553516 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.553476 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:50:44.553560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.553515 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:50:44.679310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.679271 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:44.884321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:44.884237 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:45.561260 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:45.561220 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" event={"ID":"03da32a5-06a1-46e4-8379-154d9524ad4d","Type":"ContainerStarted","Data":"8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2"} Nov 02 21:50:46.406474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:46.406431 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:46.484989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:46.482230 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:50:46.499046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:46.495058 4441 patch_prober.go:29] interesting pod/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 21:50:46.499046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:46.495128 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 21:50:46.549878 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:46.549843 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" podUID="0f385b13-7059-40f1-876f-d9f0a0eaf369" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:50:46.953992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:46.953281 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:47.180065 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:47.172890 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:47.486989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:47.483236 4441 patch_prober.go:29] interesting pod/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 21:50:47.486989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:47.483306 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 21:50:47.616067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:47.612192 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:47.932997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:47.927223 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:48.033426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.033381 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb"] Nov 02 21:50:48.038095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.038074 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.093481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.093435 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb"] Nov 02 21:50:48.120389 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.120355 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213a55c1-c58b-43bc-b2e2-222ea25f9d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.123334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.123301 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2wx\" (UniqueName: \"kubernetes.io/projected/213a55c1-c58b-43bc-b2e2-222ea25f9d78-kube-api-access-kg2wx\") pod \"payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.125956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.125931 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d4f799-dockercfg-brlvb\"" Nov 02 21:50:48.126530 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.126255 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d4f799\"" Nov 02 21:50:48.267052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.253945 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213a55c1-c58b-43bc-b2e2-222ea25f9d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.301378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.301335 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2wx\" (UniqueName: \"kubernetes.io/projected/213a55c1-c58b-43bc-b2e2-222ea25f9d78-kube-api-access-kg2wx\") pod \"payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.322453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.268097 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213a55c1-c58b-43bc-b2e2-222ea25f9d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.348917 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.348883 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg"] Nov 02 21:50:48.353386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.353359 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.355047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.354834 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg"] Nov 02 21:50:48.399316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.399290 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2wx\" (UniqueName: \"kubernetes.io/projected/213a55c1-c58b-43bc-b2e2-222ea25f9d78-kube-api-access-kg2wx\") pod \"payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.423705 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.423676 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/dad61665-5a6c-4452-a0b5-32b8cd27308f-kube-api-access-zv26d\") pod \"payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.423887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.423746 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad61665-5a6c-4452-a0b5-32b8cd27308f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.495140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.495106 4441 patch_prober.go:29] interesting pod/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 21:50:48.495308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.495161 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 21:50:48.524844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.524771 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/dad61665-5a6c-4452-a0b5-32b8cd27308f-kube-api-access-zv26d\") pod \"payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.524844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.524838 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad61665-5a6c-4452-a0b5-32b8cd27308f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.525579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.525553 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad61665-5a6c-4452-a0b5-32b8cd27308f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.601420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.601384 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/dad61665-5a6c-4452-a0b5-32b8cd27308f-kube-api-access-zv26d\") pod \"payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.624454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.624425 4441 generic.go:356] "Generic (PLEG): container finished" podID="0f385b13-7059-40f1-876f-d9f0a0eaf369" containerID="9ef8454f649a9e2b517d0f086edf76e3757f2e96b99d607bac9eb640031ced14" exitCode=0 Nov 02 21:50:48.624630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.624499 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" event={"ID":"0f385b13-7059-40f1-876f-d9f0a0eaf369","Type":"ContainerDied","Data":"9ef8454f649a9e2b517d0f086edf76e3757f2e96b99d607bac9eb640031ced14"} Nov 02 21:50:48.656222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.654207 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:48.704871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.704834 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:48.743724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.743702 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:50:48.828427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.828350 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f385b13-7059-40f1-876f-d9f0a0eaf369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f385b13-7059-40f1-876f-d9f0a0eaf369-plugins-config-volume\") pod \"0f385b13-7059-40f1-876f-d9f0a0eaf369\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " Nov 02 21:50:48.828689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.828595 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f385b13-7059-40f1-876f-d9f0a0eaf369-kube-api-access-mrmx7\" (UniqueName: \"kubernetes.io/projected/0f385b13-7059-40f1-876f-d9f0a0eaf369-kube-api-access-mrmx7\") pod \"0f385b13-7059-40f1-876f-d9f0a0eaf369\" (UID: \"0f385b13-7059-40f1-876f-d9f0a0eaf369\") " Nov 02 21:50:48.830012 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.829875 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f385b13-7059-40f1-876f-d9f0a0eaf369-plugins-config-volume" pod "0f385b13-7059-40f1-876f-d9f0a0eaf369" (UID: "0f385b13-7059-40f1-876f-d9f0a0eaf369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:48.832574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.832504 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f385b13-7059-40f1-876f-d9f0a0eaf369-kube-api-access-mrmx7" pod "0f385b13-7059-40f1-876f-d9f0a0eaf369" (UID: "0f385b13-7059-40f1-876f-d9f0a0eaf369"). InnerVolumeSpecName "kube-api-access-mrmx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:48.931141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.931080 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f385b13-7059-40f1-876f-d9f0a0eaf369-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:48.931141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:48.931115 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrmx7\" (UniqueName: \"kubernetes.io/projected/0f385b13-7059-40f1-876f-d9f0a0eaf369-kube-api-access-mrmx7\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:49.054045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.053687 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh"] Nov 02 21:50:49.057599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.056239 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35031903-ccwvh"] Nov 02 21:50:49.129415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.129386 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:50:49.151488 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:49.151460 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213a55c1_c58b_43bc_b2e2_222ea25f9d78.slice/crio-5774b1467e519f41459df20e561d02c39923a354f4304b2c4eaf86212e551f82 WatchSource:0}: Error finding container 5774b1467e519f41459df20e561d02c39923a354f4304b2c4eaf86212e551f82: Status 404 returned error can't find the container with id 5774b1467e519f41459df20e561d02c39923a354f4304b2c4eaf86212e551f82 Nov 02 21:50:49.152836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.152817 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb"] Nov 02 21:50:49.200591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.200545 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg"] Nov 02 21:50:49.201052 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:49.201021 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad61665_5a6c_4452_a0b5_32b8cd27308f.slice/crio-166c19fdc60b0caa60cc6018bfb8e5b382ab3aa085e6bd1611ba8800f409b86f WatchSource:0}: Error finding container 166c19fdc60b0caa60cc6018bfb8e5b382ab3aa085e6bd1611ba8800f409b86f: Status 404 returned error can't find the container with id 166c19fdc60b0caa60cc6018bfb8e5b382ab3aa085e6bd1611ba8800f409b86f Nov 02 21:50:49.480308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.480274 4441 patch_prober.go:29] interesting pod/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 21:50:49.480588 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.480550 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 21:50:49.634611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.634572 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" event={"ID":"213a55c1-c58b-43bc-b2e2-222ea25f9d78","Type":"ContainerStarted","Data":"ac11e13931dbf956c7181015351243dac43cd1d0431b6d257d0c57e3741b3ce8"} Nov 02 21:50:49.634802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.634633 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" event={"ID":"213a55c1-c58b-43bc-b2e2-222ea25f9d78","Type":"ContainerStarted","Data":"5774b1467e519f41459df20e561d02c39923a354f4304b2c4eaf86212e551f82"} Nov 02 21:50:49.634802 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.634683 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:50:49.636247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.636223 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" event={"ID":"0f385b13-7059-40f1-876f-d9f0a0eaf369","Type":"ContainerDied","Data":"68d43c514685dfaf4a4eaae4b9f1e150f790e8c969c5f09c41ec1bd0c1a8aeea"} Nov 02 21:50:49.636390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.636255 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s" Nov 02 21:50:49.636390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.636274 4441 scope.go:122] "RemoveContainer" containerID="9ef8454f649a9e2b517d0f086edf76e3757f2e96b99d607bac9eb640031ced14" Nov 02 21:50:49.639055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.638955 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" event={"ID":"dad61665-5a6c-4452-a0b5-32b8cd27308f","Type":"ContainerStarted","Data":"434625c44f9a6326f22cf66ccbfd3325a28e3c191b1fbaec7d1109f4a0e56b59"} Nov 02 21:50:49.639223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.639209 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:50:49.639345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.639328 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" event={"ID":"dad61665-5a6c-4452-a0b5-32b8cd27308f","Type":"ContainerStarted","Data":"166c19fdc60b0caa60cc6018bfb8e5b382ab3aa085e6bd1611ba8800f409b86f"} Nov 02 21:50:49.748333 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.748227 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" podStartSLOduration=2.74820698 podStartE2EDuration="2.74820698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:49.74432695 +0000 UTC m=+2279.299955287" watchObservedRunningTime="2026-11-02 21:50:49.74820698 +0000 UTC m=+2279.303835320" Nov 02 21:50:49.848438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.848410 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:49.855597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.855459 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" podStartSLOduration=1.855442942 podStartE2EDuration="1.855442942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:49.853401662 +0000 UTC m=+2279.409029997" watchObservedRunningTime="2026-11-02 21:50:49.855442942 +0000 UTC m=+2279.411071280" Nov 02 21:50:49.965615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.965586 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s"] Nov 02 21:50:49.969999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:49.969879 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dd6574-5fd9c947b7-x7c6s"] Nov 02 21:50:50.000622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:50.000490 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:50.131936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:50.131805 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:50.480450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:50.480416 4441 patch_prober.go:29] interesting pod/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Nov 02 21:50:50.480743 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:50.480715 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Nov 02 21:50:50.603784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:50.603744 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f385b13-7059-40f1-876f-d9f0a0eaf369" path="/var/lib/kubelet/pods/0f385b13-7059-40f1-876f-d9f0a0eaf369/volumes" Nov 02 21:50:50.604400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:50.604377 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df719e93-4ca7-4a6b-a317-aeab83d56223" path="/var/lib/kubelet/pods/df719e93-4ca7-4a6b-a317-aeab83d56223/volumes" Nov 02 21:50:51.486925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:51.486896 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:50:51.488112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:51.488081 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:51.488242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:51.488198 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 21:50:51.689581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:51.689552 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:51.820691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:51.820661 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:53.249257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:53.249224 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:53.624534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:53.624503 4441 scope.go:122] "RemoveContainer" containerID="98a7a5710778d3496e1ec261419a32e97e230aaf3e4843fee308c98ca0ed134f" Nov 02 21:50:54.046224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:54.046084 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:54.206569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:54.206530 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:54.336153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:54.336122 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:55.116669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:55.116629 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:56.070669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:56.070637 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:56.272347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:56.272309 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:56.422491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:56.422388 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:56.660101 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:56.660060 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk"] Nov 02 21:50:56.708291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:56.708216 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b983e02-fmhbk"] Nov 02 21:50:56.908122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:56.908091 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:57.042046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.039667 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:57.165042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.165011 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:57.329396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.329358 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp"] Nov 02 21:50:57.330575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.330545 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0f385b13-7059-40f1-876f-d9f0a0eaf369" containerName="payload-pre-processing" Nov 02 21:50:57.330575 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.330574 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f385b13-7059-40f1-876f-d9f0a0eaf369" containerName="payload-pre-processing" Nov 02 21:50:57.330754 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.330717 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f385b13-7059-40f1-876f-d9f0a0eaf369" containerName="payload-pre-processing" Nov 02 21:50:57.340008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.338704 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.382928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.382889 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp"] Nov 02 21:50:57.397815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.395896 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3566f9\"" Nov 02 21:50:57.401715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.399355 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb20a5f-ae15-4d73-be8e-e56145047134-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.401715 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.399578 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfbj9\" (UniqueName: \"kubernetes.io/projected/3fb20a5f-ae15-4d73-be8e-e56145047134-kube-api-access-bfbj9\") pod \"payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.427010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.424254 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3566f9-dockercfg-w97cc\"" Nov 02 21:50:57.572039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.569205 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb20a5f-ae15-4d73-be8e-e56145047134-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.572039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.569440 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfbj9\" (UniqueName: \"kubernetes.io/projected/3fb20a5f-ae15-4d73-be8e-e56145047134-kube-api-access-bfbj9\") pod \"payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.572039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.570817 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb20a5f-ae15-4d73-be8e-e56145047134-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.616109 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.616028 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf"] Nov 02 21:50:57.623026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.623004 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:57.629910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.629885 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf"] Nov 02 21:50:57.635046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.631891 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfbj9\" (UniqueName: \"kubernetes.io/projected/3fb20a5f-ae15-4d73-be8e-e56145047134-kube-api-access-bfbj9\") pod \"payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.658061 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.658037 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:57.670651 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.670620 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrdd\" (UniqueName: \"kubernetes.io/projected/d251ed63-5ecc-4824-b440-4c7d5f363ff3-kube-api-access-glrdd\") pod \"payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:57.670833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.670810 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d251ed63-5ecc-4824-b440-4c7d5f363ff3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:57.773011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.772123 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d251ed63-5ecc-4824-b440-4c7d5f363ff3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:57.773011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.772325 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrdd\" (UniqueName: \"kubernetes.io/projected/d251ed63-5ecc-4824-b440-4c7d5f363ff3-kube-api-access-glrdd\") pod \"payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:57.773506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.773484 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d251ed63-5ecc-4824-b440-4c7d5f363ff3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:57.827038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.827009 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrdd\" (UniqueName: \"kubernetes.io/projected/d251ed63-5ecc-4824-b440-4c7d5f363ff3-kube-api-access-glrdd\") pod \"payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:57.955137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:57.955067 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:58.034860 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:58.034814 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb20a5f_ae15_4d73_be8e_e56145047134.slice/crio-b0f85d578291a22bb8b24e3232c43124ef0646b677f13fe607c9975307a6b68b WatchSource:0}: Error finding container b0f85d578291a22bb8b24e3232c43124ef0646b677f13fe607c9975307a6b68b: Status 404 returned error can't find the container with id b0f85d578291a22bb8b24e3232c43124ef0646b677f13fe607c9975307a6b68b Nov 02 21:50:58.035429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.035409 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp"] Nov 02 21:50:58.337286 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:50:58.337256 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd251ed63_5ecc_4824_b440_4c7d5f363ff3.slice/crio-5d611b64adb2b50da0e6fd7f93562078ac9638eac2a986e6caa08ded98bea386 WatchSource:0}: Error finding container 5d611b64adb2b50da0e6fd7f93562078ac9638eac2a986e6caa08ded98bea386: Status 404 returned error can't find the container with id 5d611b64adb2b50da0e6fd7f93562078ac9638eac2a986e6caa08ded98bea386 Nov 02 21:50:58.338398 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.338373 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf"] Nov 02 21:50:58.589542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.589449 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c750bd06-fefd-4705-8136-b1804084625d" path="/var/lib/kubelet/pods/c750bd06-fefd-4705-8136-b1804084625d/volumes" Nov 02 21:50:58.721202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.721165 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" event={"ID":"3fb20a5f-ae15-4d73-be8e-e56145047134","Type":"ContainerStarted","Data":"503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a"} Nov 02 21:50:58.721427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.721231 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" event={"ID":"3fb20a5f-ae15-4d73-be8e-e56145047134","Type":"ContainerStarted","Data":"b0f85d578291a22bb8b24e3232c43124ef0646b677f13fe607c9975307a6b68b"} Nov 02 21:50:58.721427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.721262 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:50:58.722979 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.722944 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" event={"ID":"d251ed63-5ecc-4824-b440-4c7d5f363ff3","Type":"ContainerStarted","Data":"b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39"} Nov 02 21:50:58.723074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.722994 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" event={"ID":"d251ed63-5ecc-4824-b440-4c7d5f363ff3","Type":"ContainerStarted","Data":"5d611b64adb2b50da0e6fd7f93562078ac9638eac2a986e6caa08ded98bea386"} Nov 02 21:50:58.723074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.723017 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:50:58.832274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.832224 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" podStartSLOduration=1.832211313 podStartE2EDuration="1.832211313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:58.829097409 +0000 UTC m=+2288.384725746" watchObservedRunningTime="2026-11-02 21:50:58.832211313 +0000 UTC m=+2288.387839651" Nov 02 21:50:58.962039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:50:58.959195 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" podStartSLOduration=1.9591683930000001 podStartE2EDuration="1.959168393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:58.935827945 +0000 UTC m=+2288.491456285" watchObservedRunningTime="2026-11-02 21:50:58.959168393 +0000 UTC m=+2288.514796733" Nov 02 21:51:00.483515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:00.483483 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:00.618206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:00.618173 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:00.649612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:00.649585 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 21:51:00.650017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:00.650000 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 21:51:01.305075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:01.305044 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:02.088262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:02.088162 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:06.087986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:06.087933 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:09.734573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:09.734530 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 21:51:09.735070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:09.734681 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 21:51:10.615306 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:51:10.615272 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:51:11.004805 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:11.004733 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:15.414411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:15.414369 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:15.569231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:15.569174 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:15.725470 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:15.725375 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:20.209594 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:20.209555 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:24.801460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:24.801425 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:29.411778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:29.411745 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:33.278604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:33.278550 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:34.512486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:34.512452 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:34.764724 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:34.764635 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:35.061026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:35.060959 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:35.275106 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:35.275065 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:35.278692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:35.278663 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd"] Nov 02 21:51:35.325309 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:35.325239 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dd6574-gs7bd"] Nov 02 21:51:35.664042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:35.663943 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:36.584920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:36.584886 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11190fa-b0e0-493d-b72a-be9883c6f515" path="/var/lib/kubelet/pods/d11190fa-b0e0-493d-b72a-be9883c6f515/volumes" Nov 02 21:51:38.345474 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:38.345441 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:39.044036 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:39.044003 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj"] Nov 02 21:51:39.091135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:39.091107 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15edb8c5-w25mj"] Nov 02 21:51:39.465091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:39.465063 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:40.075092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:40.075049 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:40.585287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:40.585256 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc4360d-0e1c-422f-b176-a4475691450c" path="/var/lib/kubelet/pods/1bc4360d-0e1c-422f-b176-a4475691450c/volumes" Nov 02 21:51:40.616531 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:51:40.616496 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:51:41.383762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:41.383730 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:41.529050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:41.529013 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:41.720909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:41.720825 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:41.741792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:41.741761 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp"] Nov 02 21:51:41.742034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:41.742011 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" containerID="cri-o://93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384" gracePeriod=30 Nov 02 21:51:41.864684 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:41.864644 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:44.843074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:44.843039 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:45.135122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:45.135046 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:46.662218 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:46.662179 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:47.039321 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.039293 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:51:47.121567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121538 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-certs\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.121750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121593 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-socket\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.121750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121639 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-podinfo\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.121750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121678 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-envoy\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.121750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121710 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8430cd15-771a-4ace-85d9-6589bbd0ad57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430cd15-771a-4ace-85d9-6589bbd0ad57-istiod-ca-cert\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.122016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121809 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-token\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-token\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.122016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121902 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-credential-socket\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.122016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121937 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-kube-api-access-99tkc\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-kube-api-access-99tkc\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.122016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121849 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-certs" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.122016 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.121944 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-socket" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.122308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.122194 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-credential-socket" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.122308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.122291 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8430cd15-771a-4ace-85d9-6589bbd0ad57-istiod-ca-cert" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:47.122536 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.122517 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-data\") pod \"8430cd15-771a-4ace-85d9-6589bbd0ad57\" (UID: \"8430cd15-771a-4ace-85d9-6589bbd0ad57\") " Nov 02 21:51:47.122794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.122760 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-data" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.123354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.123329 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.123465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.123358 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.123465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.123374 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.123465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.123389 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.123465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.123403 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8430cd15-771a-4ace-85d9-6589bbd0ad57-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.124491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.124454 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-token" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:47.124602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.124488 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-envoy" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.124665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.124638 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-kube-api-access-99tkc" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "kube-api-access-99tkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:47.124727 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.124690 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-podinfo" pod "8430cd15-771a-4ace-85d9-6589bbd0ad57" (UID: "8430cd15-771a-4ace-85d9-6589bbd0ad57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:51:47.186760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.186609 4441 generic.go:356] "Generic (PLEG): container finished" podID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerID="93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384" exitCode=0 Nov 02 21:51:47.186760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.186672 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" event={"ID":"8430cd15-771a-4ace-85d9-6589bbd0ad57","Type":"ContainerDied","Data":"93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384"} Nov 02 21:51:47.186760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.186735 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" Nov 02 21:51:47.186760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.186751 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp" event={"ID":"8430cd15-771a-4ace-85d9-6589bbd0ad57","Type":"ContainerDied","Data":"783e94d38dcdfcaeaa8a05a6ccbff022267b9cd6af701965f727866b7d0342e9"} Nov 02 21:51:47.187404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.186781 4441 scope.go:122] "RemoveContainer" containerID="93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384" Nov 02 21:51:47.210823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.210800 4441 scope.go:122] "RemoveContainer" containerID="80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7" Nov 02 21:51:47.223526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.223507 4441 scope.go:122] "RemoveContainer" containerID="93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384" Nov 02 21:51:47.223874 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:51:47.223848 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384\": container with ID starting with 93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384 not found: ID does not exist" containerID="93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384" Nov 02 21:51:47.224007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.223888 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384"} err="failed to get container status \"93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384\": rpc error: code = NotFound desc = could not find container \"93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384\": container with ID starting with 93dec61809cf68c67173dcff2f06af6eb5df18430e9e7ec08736c5fb2b1c0384 not found: ID does not exist" Nov 02 21:51:47.224007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.223914 4441 scope.go:122] "RemoveContainer" containerID="80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7" Nov 02 21:51:47.224277 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:51:47.224252 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7\": container with ID starting with 80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7 not found: ID does not exist" containerID="80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7" Nov 02 21:51:47.224323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.224287 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7"} err="failed to get container status \"80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7\": rpc error: code = NotFound desc = could not find container \"80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7\": container with ID starting with 80c713d19e9708a809b8999c8de0ff69e6a8679ea3071762fedf71a67c9d2cb7 not found: ID does not exist" Nov 02 21:51:47.224836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.224814 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.224921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.224840 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.224921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.224855 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.224921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.224868 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tkc\" (UniqueName: \"kubernetes.io/projected/8430cd15-771a-4ace-85d9-6589bbd0ad57-kube-api-access-99tkc\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.355157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.355126 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp"] Nov 02 21:51:47.359146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.359119 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-255tp"] Nov 02 21:51:47.489082 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:47.489004 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:48.254416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:48.254376 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:48.424992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:48.422312 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:48.586915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:48.586871 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" path="/var/lib/kubelet/pods/8430cd15-771a-4ace-85d9-6589bbd0ad57/volumes" Nov 02 21:51:49.653922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:49.653890 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:49.808453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:49.808414 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:49.928187 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:49.928106 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:52.381136 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:52.381096 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:52.583528 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:52.583488 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:52.740055 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:52.738755 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:53.607382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:53.607346 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:53.696661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:53.696628 4441 scope.go:122] "RemoveContainer" containerID="9943101b88c65fae8b39513d40a9961d5767bdda9d93693b36932e60a12fd14d" Nov 02 21:51:53.707217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:53.707198 4441 scope.go:122] "RemoveContainer" containerID="bcae2283734f3ebb89405917211c3ccd129db97ab84605fa2581d509fd8f8864" Nov 02 21:51:53.716823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:53.716803 4441 scope.go:122] "RemoveContainer" containerID="95c7e699b2c1c60b22b7d48e31c71572539a5d39910763a7321fe5ec793af3fd" Nov 02 21:51:53.725759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:53.725738 4441 scope.go:122] "RemoveContainer" containerID="15b03cbef65bd4774e36c29ee2f515fd082dc277d2cfaf576c1358f6076b569d" Nov 02 21:51:54.894045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:54.893947 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:55.519352 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.519317 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:55.723815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.723782 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:55.934507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.934463 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf"] Nov 02 21:51:55.935171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.935150 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" Nov 02 21:51:55.935171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.935169 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" Nov 02 21:51:55.935334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.935196 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" Nov 02 21:51:55.935334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.935204 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" Nov 02 21:51:55.935334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.935311 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" Nov 02 21:51:55.940643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.940620 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:55.991337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:55.991307 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf"] Nov 02 21:51:56.026355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.026309 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45042402-da10-4f5d-8bfd-05b3e36ba5b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:56.026639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.026614 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mrw\" (UniqueName: \"kubernetes.io/projected/45042402-da10-4f5d-8bfd-05b3e36ba5b4-kube-api-access-z5mrw\") pod \"payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:56.032659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.032631 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3d2b29-dockercfg-lm7l6\"" Nov 02 21:51:56.032808 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.032719 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3d2b29\"" Nov 02 21:51:56.134009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.128191 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mrw\" (UniqueName: \"kubernetes.io/projected/45042402-da10-4f5d-8bfd-05b3e36ba5b4-kube-api-access-z5mrw\") pod \"payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:56.134009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.128301 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45042402-da10-4f5d-8bfd-05b3e36ba5b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:56.134009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.129367 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45042402-da10-4f5d-8bfd-05b3e36ba5b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:56.197083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.192183 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mrw\" (UniqueName: \"kubernetes.io/projected/45042402-da10-4f5d-8bfd-05b3e36ba5b4-kube-api-access-z5mrw\") pod \"payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:56.260060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.260024 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:56.634349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:56.633090 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf"] Nov 02 21:51:56.634349 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:51:56.633600 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45042402_da10_4f5d_8bfd_05b3e36ba5b4.slice/crio-09940b307521f94341a8cb616d72e95baa23d2db5354398befd996091f089eca WatchSource:0}: Error finding container 09940b307521f94341a8cb616d72e95baa23d2db5354398befd996091f089eca: Status 404 returned error can't find the container with id 09940b307521f94341a8cb616d72e95baa23d2db5354398befd996091f089eca Nov 02 21:51:57.285864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:57.285826 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" event={"ID":"45042402-da10-4f5d-8bfd-05b3e36ba5b4","Type":"ContainerStarted","Data":"1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3"} Nov 02 21:51:57.286525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:57.285920 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" event={"ID":"45042402-da10-4f5d-8bfd-05b3e36ba5b4","Type":"ContainerStarted","Data":"09940b307521f94341a8cb616d72e95baa23d2db5354398befd996091f089eca"} Nov 02 21:51:57.286525 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:57.285955 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:51:57.393248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:57.393187 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" podStartSLOduration=2.393170266 podStartE2EDuration="2.393170266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:57.39187511 +0000 UTC m=+2346.947503448" watchObservedRunningTime="2026-11-02 21:51:57.393170266 +0000 UTC m=+2346.948798604" Nov 02 21:51:59.737386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:59.737357 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:59.892547 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:51:59.892512 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:00.013502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:00.013400 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:04.398623 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:04.398590 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.016397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.016346 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.144995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.144933 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.308048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.307137 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:52:08.430010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.429700 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.467147 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.467109 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv"] Nov 02 21:52:08.467792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.467759 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" containerID="cri-o://0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530" gracePeriod=30 Nov 02 21:52:08.484626 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.484602 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl"] Nov 02 21:52:08.485005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.484961 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" containerID="cri-o://0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6" gracePeriod=30 Nov 02 21:52:08.492541 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.492516 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw"] Nov 02 21:52:08.492921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:08.492877 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" containerID="cri-o://2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065" gracePeriod=30 Nov 02 21:52:09.404573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:09.404534 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:10.614707 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:10.614674 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:52:12.785245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:12.785210 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:13.220665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.220630 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:13.718739 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.718709 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw"] Nov 02 21:52:13.720077 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.720054 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8430cd15-771a-4ace-85d9-6589bbd0ad57" containerName="istio-proxy" Nov 02 21:52:13.729807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.729143 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.805824 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-credential-socket\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.805907 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-token\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.805991 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-socket\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.806024 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-certs\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.806280 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-podinfo\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.806327 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kj6\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-kube-api-access-69kj6\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.806402 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee9-3056-48a2-bed0-19b609cb1795-istiod-ca-cert\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.806436 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-data\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.807314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.806480 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-envoy\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.849269 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.848492 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:13.849683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.849644 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-11537e-openshift-default-dockercfg-5kkj8\"" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.898056 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw"] Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.911908 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-socket\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.911957 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-certs\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.912174 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-podinfo\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.912218 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kj6\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-kube-api-access-69kj6\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.912294 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee9-3056-48a2-bed0-19b609cb1795-istiod-ca-cert\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.912331 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-data\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.912379 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-envoy\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.918909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.912422 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-credential-socket\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.940370 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.940002 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-socket\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.940855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.940822 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-token\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.947387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.945861 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-data\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.947862 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.947837 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-certs\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.957206 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.949297 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-credential-socket\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.968659 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.965441 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-envoy\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.968824 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.966392 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-podinfo\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:13.969221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:13.969179 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee9-3056-48a2-bed0-19b609cb1795-istiod-ca-cert\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:14.004054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.004025 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kj6\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-kube-api-access-69kj6\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:14.007775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.006449 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-token\") pod \"e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:14.032841 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.032788 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:52:14.044664 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.044616 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:52:14.047150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.047126 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:52:14.058288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.058263 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:14.149744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.140326 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161099 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-data\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161156 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-certs\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161181 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16840db4-8c59-4fbf-87a6-24a83d2b379b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16840db4-8c59-4fbf-87a6-24a83d2b379b-istiod-ca-cert\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161206 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-certs\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161243 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-podinfo\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161268 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-data\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161286 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6604492-7b11-491c-87b9-dc752ba35206-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6604492-7b11-491c-87b9-dc752ba35206-istiod-ca-cert\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161320 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-data\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161359 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6604492-7b11-491c-87b9-dc752ba35206-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6604492-7b11-491c-87b9-dc752ba35206-istio-podinfo\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161381 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-credential-socket\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161407 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-envoy\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161427 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-socket\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161448 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-envoy\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161491 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74df9aca-99d0-457b-b866-9e0ca409b0b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9aca-99d0-457b-b866-9e0ca409b0b3-istiod-ca-cert\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161557 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-token\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-token\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161618 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-istio-token\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-istio-token\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161660 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-socket\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161681 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-kube-api-access-xklht\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-kube-api-access-xklht\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161709 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-socket\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161740 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-credential-socket\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.161775 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161796 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-certs\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.162616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161820 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-token\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-token\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.162616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161838 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-kube-api-access-jp8wx\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-kube-api-access-jp8wx\") pod \"e6604492-7b11-491c-87b9-dc752ba35206\" (UID: \"e6604492-7b11-491c-87b9-dc752ba35206\") " Nov 02 21:52:14.162616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161862 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-envoy\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.162616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161890 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-podinfo\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.162616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161912 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-kube-api-access-4cp4k\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-kube-api-access-4cp4k\") pod \"74df9aca-99d0-457b-b866-9e0ca409b0b3\" (UID: \"74df9aca-99d0-457b-b866-9e0ca409b0b3\") " Nov 02 21:52:14.162616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.161943 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-credential-socket\") pod \"16840db4-8c59-4fbf-87a6-24a83d2b379b\" (UID: \"16840db4-8c59-4fbf-87a6-24a83d2b379b\") " Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.163289 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-data" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.163524 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-credential-socket" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.163595 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-data" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.164055 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-certs" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.164096 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-certs" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.164105 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16840db4-8c59-4fbf-87a6-24a83d2b379b-istiod-ca-cert" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.164410 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-certs" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.165025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.164485 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74df9aca-99d0-457b-b866-9e0ca409b0b3-istiod-ca-cert" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:14.166798 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.166761 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-credential-socket" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.168409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.168382 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-envoy" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.168409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.168390 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-podinfo" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:14.168643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.168623 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-credential-socket" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.169008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.168955 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-socket" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.169720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.169526 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-kube-api-access-jp8wx" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "kube-api-access-jp8wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:14.170126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.169863 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-socket" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.171332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.171076 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-token" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:14.171777 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.171691 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-token" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:14.172388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.172203 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-socket" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.172812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.172605 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6604492-7b11-491c-87b9-dc752ba35206-istiod-ca-cert" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:14.173162 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.172948 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-data" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.173162 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.173079 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-podinfo" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:14.174156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.174125 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-istio-token" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:14.174665 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.174633 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-kube-api-access-xklht" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "kube-api-access-xklht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:14.175612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.175523 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-kube-api-access-4cp4k" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "kube-api-access-4cp4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:14.175612 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.175583 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6604492-7b11-491c-87b9-dc752ba35206-istio-podinfo" pod "e6604492-7b11-491c-87b9-dc752ba35206" (UID: "e6604492-7b11-491c-87b9-dc752ba35206"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:14.176426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.176380 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-envoy" pod "16840db4-8c59-4fbf-87a6-24a83d2b379b" (UID: "16840db4-8c59-4fbf-87a6-24a83d2b379b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.176573 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.176543 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-envoy" pod "74df9aca-99d0-457b-b866-9e0ca409b0b3" (UID: "74df9aca-99d0-457b-b866-9e0ca409b0b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:14.263940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.263904 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6604492-7b11-491c-87b9-dc752ba35206-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.263940 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.263936 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.263952 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.263998 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264014 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264025 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9aca-99d0-457b-b866-9e0ca409b0b3-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264036 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264051 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264063 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264076 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xklht\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-kube-api-access-xklht\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264091 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264107 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264119 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264134 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264146 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8wx\" (UniqueName: \"kubernetes.io/projected/e6604492-7b11-491c-87b9-dc752ba35206-kube-api-access-jp8wx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264159 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264173 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264185 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cp4k\" (UniqueName: \"kubernetes.io/projected/74df9aca-99d0-457b-b866-9e0ca409b0b3-kube-api-access-4cp4k\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264200 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264215 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264229 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264241 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16840db4-8c59-4fbf-87a6-24a83d2b379b-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264255 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264268 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9aca-99d0-457b-b866-9e0ca409b0b3-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264282 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16840db4-8c59-4fbf-87a6-24a83d2b379b-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264295 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6604492-7b11-491c-87b9-dc752ba35206-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.264907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.264309 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6604492-7b11-491c-87b9-dc752ba35206-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.484698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.484667 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw"] Nov 02 21:52:14.485331 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:52:14.485305 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e865ee9_3056_48a2_bed0_19b609cb1795.slice/crio-00eff63ede474ecf3f59fd1512d7f957f5e926dbd6e5910fe2b0dee275a2a8c0 WatchSource:0}: Error finding container 00eff63ede474ecf3f59fd1512d7f957f5e926dbd6e5910fe2b0dee275a2a8c0: Status 404 returned error can't find the container with id 00eff63ede474ecf3f59fd1512d7f957f5e926dbd6e5910fe2b0dee275a2a8c0 Nov 02 21:52:14.488558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.488525 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:14.488654 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.488609 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:14.488713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.488655 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:14.529460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.529429 4441 generic.go:356] "Generic (PLEG): container finished" podID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerID="2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065" exitCode=0 Nov 02 21:52:14.529590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.529472 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" Nov 02 21:52:14.529590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.529510 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" event={"ID":"16840db4-8c59-4fbf-87a6-24a83d2b379b","Type":"ContainerDied","Data":"2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065"} Nov 02 21:52:14.529590 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.529568 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw" event={"ID":"16840db4-8c59-4fbf-87a6-24a83d2b379b","Type":"ContainerDied","Data":"4a33cd3fa8b2ccc91993705447163ff964618220cd294cddfb8c0424434ee423"} Nov 02 21:52:14.529757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.529597 4441 scope.go:122] "RemoveContainer" containerID="2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065" Nov 02 21:52:14.531002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.530956 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" event={"ID":"3e865ee9-3056-48a2-bed0-19b609cb1795","Type":"ContainerStarted","Data":"00eff63ede474ecf3f59fd1512d7f957f5e926dbd6e5910fe2b0dee275a2a8c0"} Nov 02 21:52:14.532672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.532651 4441 generic.go:356] "Generic (PLEG): container finished" podID="e6604492-7b11-491c-87b9-dc752ba35206" containerID="0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6" exitCode=0 Nov 02 21:52:14.532815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.532798 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" event={"ID":"e6604492-7b11-491c-87b9-dc752ba35206","Type":"ContainerDied","Data":"0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6"} Nov 02 21:52:14.532891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.532842 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" event={"ID":"e6604492-7b11-491c-87b9-dc752ba35206","Type":"ContainerDied","Data":"cf88d61454c6807a4e043e239d70edbcd32e2abebac646a83d679bc960052f0d"} Nov 02 21:52:14.532950 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.532922 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl" Nov 02 21:52:14.537814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.537790 4441 generic.go:356] "Generic (PLEG): container finished" podID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerID="0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530" exitCode=0 Nov 02 21:52:14.537926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.537844 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" Nov 02 21:52:14.537926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.537867 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" event={"ID":"74df9aca-99d0-457b-b866-9e0ca409b0b3","Type":"ContainerDied","Data":"0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530"} Nov 02 21:52:14.537926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.537902 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv" event={"ID":"74df9aca-99d0-457b-b866-9e0ca409b0b3","Type":"ContainerDied","Data":"c9037edd9084a1311f879274ce8f1d3c1ca2393f18918a093229d253e3adbac4"} Nov 02 21:52:14.543954 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.543939 4441 scope.go:122] "RemoveContainer" containerID="2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065" Nov 02 21:52:14.544235 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:14.544216 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065\": container with ID starting with 2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065 not found: ID does not exist" containerID="2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065" Nov 02 21:52:14.544302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.544244 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065"} err="failed to get container status \"2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065\": rpc error: code = NotFound desc = could not find container \"2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065\": container with ID starting with 2d09ebc085d4ec3767dcf9afbb8e4335d458e86d74a0cfad925449b9fe713065 not found: ID does not exist" Nov 02 21:52:14.544302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.544260 4441 scope.go:122] "RemoveContainer" containerID="0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6" Nov 02 21:52:14.555570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.555554 4441 scope.go:122] "RemoveContainer" containerID="0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6" Nov 02 21:52:14.555807 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:14.555789 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6\": container with ID starting with 0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6 not found: ID does not exist" containerID="0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6" Nov 02 21:52:14.555885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.555813 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6"} err="failed to get container status \"0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6\": rpc error: code = NotFound desc = could not find container \"0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6\": container with ID starting with 0fc897838760b1896424067a4e254646208c04bd2ca37fe792951a3c1934c6e6 not found: ID does not exist" Nov 02 21:52:14.555885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.555827 4441 scope.go:122] "RemoveContainer" containerID="0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530" Nov 02 21:52:14.566152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.566137 4441 scope.go:122] "RemoveContainer" containerID="0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530" Nov 02 21:52:14.566403 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:14.566383 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530\": container with ID starting with 0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530 not found: ID does not exist" containerID="0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530" Nov 02 21:52:14.566454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.566409 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530"} err="failed to get container status \"0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530\": rpc error: code = NotFound desc = could not find container \"0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530\": container with ID starting with 0e0e7330364051cfe938dcd7b6c37c696cc1b6004170f98fd443935575657530 not found: ID does not exist" Nov 02 21:52:14.697469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.697437 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl"] Nov 02 21:52:14.704163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.704125 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-p7dbl"] Nov 02 21:52:14.705283 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.705255 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:14.811640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.811613 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv"] Nov 02 21:52:14.818059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.818029 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-gdqtv"] Nov 02 21:52:14.922985 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.920068 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw"] Nov 02 21:52:14.925188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:14.925158 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-hgtfw"] Nov 02 21:52:15.547907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:15.547869 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" event={"ID":"3e865ee9-3056-48a2-bed0-19b609cb1795","Type":"ContainerStarted","Data":"cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2"} Nov 02 21:52:15.658504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:15.658449 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podStartSLOduration=2.6584351760000002 podStartE2EDuration="2.658435176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:15.656901208 +0000 UTC m=+2365.212529544" watchObservedRunningTime="2026-11-02 21:52:15.658435176 +0000 UTC m=+2365.214063513" Nov 02 21:52:16.059052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:16.059018 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:16.065001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:16.064949 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:16.558926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:16.558904 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:16.559865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:16.559846 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:52:16.586009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:16.585959 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" path="/var/lib/kubelet/pods/16840db4-8c59-4fbf-87a6-24a83d2b379b/volumes" Nov 02 21:52:16.586676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:16.586655 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" path="/var/lib/kubelet/pods/74df9aca-99d0-457b-b866-9e0ca409b0b3/volumes" Nov 02 21:52:16.587323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:16.587305 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6604492-7b11-491c-87b9-dc752ba35206" path="/var/lib/kubelet/pods/e6604492-7b11-491c-87b9-dc752ba35206/volumes" Nov 02 21:52:18.444308 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:18.444273 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:18.718660 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:18.718574 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:18.857414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:18.857376 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:19.525613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:19.525578 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:21.565066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:21.565036 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:21.745532 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:21.745492 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:24.176251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:24.176220 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:25.201552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:25.201514 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg"] Nov 02 21:52:25.202430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:25.202368 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" containerID="cri-o://cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed" gracePeriod=30 Nov 02 21:52:28.680666 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:28.680633 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:30.751302 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.751276 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:52:30.809709 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.809678 4441 generic.go:356] "Generic (PLEG): container finished" podID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerID="cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed" exitCode=0 Nov 02 21:52:30.809959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.809732 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" Nov 02 21:52:30.810070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.809753 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" event={"ID":"26245fbe-359b-4ff6-b6b0-58d816f553a1","Type":"ContainerDied","Data":"cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed"} Nov 02 21:52:30.810120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.810090 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg" event={"ID":"26245fbe-359b-4ff6-b6b0-58d816f553a1","Type":"ContainerDied","Data":"021437950f21b887f716bb3d272396cfc5b86e4d58b87f3f8fbc2a24e9e7971b"} Nov 02 21:52:30.810156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.810123 4441 scope.go:122] "RemoveContainer" containerID="cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed" Nov 02 21:52:30.829138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.828275 4441 scope.go:122] "RemoveContainer" containerID="cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed" Nov 02 21:52:30.829138 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:30.828644 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed\": container with ID starting with cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed not found: ID does not exist" containerID="cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed" Nov 02 21:52:30.829138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.828690 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed"} err="failed to get container status \"cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed\": rpc error: code = NotFound desc = could not find container \"cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed\": container with ID starting with cef8cc93329b5f8cf905d118ee703fd4da230597739102b238c5cd6076a4e8ed not found: ID does not exist" Nov 02 21:52:30.916044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.916013 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-kube-api-access-f5nvp\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-kube-api-access-f5nvp\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.916243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.916073 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-credential-socket\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.916243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.916142 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-podinfo\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.916748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.916715 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-credential-socket" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.916887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.916846 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-envoy\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.917058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.917035 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-data\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.917146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.917082 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-socket\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.917146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.917114 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-token\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-token\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.917306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.917193 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-certs\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.917306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.917231 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26245fbe-359b-4ff6-b6b0-58d816f553a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26245fbe-359b-4ff6-b6b0-58d816f553a1-istiod-ca-cert\") pod \"26245fbe-359b-4ff6-b6b0-58d816f553a1\" (UID: \"26245fbe-359b-4ff6-b6b0-58d816f553a1\") " Nov 02 21:52:30.917837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.917812 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.918202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.918177 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26245fbe-359b-4ff6-b6b0-58d816f553a1-istiod-ca-cert" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:30.919081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.919034 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-data" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.919081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.919067 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-socket" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.919268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.919217 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-certs" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.921366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.921344 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-podinfo" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:30.921493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.921418 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-token" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:30.921809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.921790 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-kube-api-access-f5nvp" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "kube-api-access-f5nvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:30.923438 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:30.923300 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-envoy" pod "26245fbe-359b-4ff6-b6b0-58d816f553a1" (UID: "26245fbe-359b-4ff6-b6b0-58d816f553a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:31.020008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.019900 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.020008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.019944 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.020008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.019959 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.020008 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.020008 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.020375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.020024 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.020375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.020040 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26245fbe-359b-4ff6-b6b0-58d816f553a1-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.020375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.020054 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26245fbe-359b-4ff6-b6b0-58d816f553a1-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.020375 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.020067 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nvp\" (UniqueName: \"kubernetes.io/projected/26245fbe-359b-4ff6-b6b0-58d816f553a1-kube-api-access-f5nvp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.411477 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.411436 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg"] Nov 02 21:52:31.451583 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:31.451542 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-2ssdg"] Nov 02 21:52:32.628219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:32.628115 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" path="/var/lib/kubelet/pods/26245fbe-359b-4ff6-b6b0-58d816f553a1/volumes" Nov 02 21:52:33.633091 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:33.633044 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": context deadline exceeded" Nov 02 21:52:33.759104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:33.759042 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:33.792095 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:33.792051 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:33.904048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:33.903460 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:33.990099 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:33.990051 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.023943 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.023861 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:34.128990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.124219 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.326054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.319529 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928"] Nov 02 21:52:34.326054 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.319769 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" containerID="cri-o://fcb0121ff601e6db2db900403079d5b5d8e9d1aaafbe1af345370e14b84212e6" gracePeriod=30 Nov 02 21:52:34.337349 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.336904 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.359009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.358834 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:34.359995 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.359262 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.368190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.367290 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2"] Nov 02 21:52:34.368190 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.367521 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" containerID="cri-o://5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e" gracePeriod=30 Nov 02 21:52:34.389032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.373142 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:52:34.389032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.376061 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:52:34.389032 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.376112 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.482992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.469097 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.482992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.470146 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.483500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.483454 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.580634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.560044 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295"] Nov 02 21:52:34.580634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.560283 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" containerID="cri-o://7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071" gracePeriod=30 Nov 02 21:52:34.787991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.771682 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.787991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.772143 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:52:34.839058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.835828 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:34.918998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.911574 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:34.922990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:34.919181 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:52:35.092050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.003228 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:35.092050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.003646 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:52:35.262992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.255277 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:35.319005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.315297 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:35.358939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.353060 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:52:35.363755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.363699 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:35.368135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.368099 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:35.495994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.471129 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:52:35.619744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.609055 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 21:52:35.619744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.609095 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:52:35.619744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.609312 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:35.644203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.632517 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 21:52:35.644203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.632901 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 02 21:52:35.868911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.868881 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:52:35.875521 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.875452 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:52:35.876010 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.875962 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 21:52:35.898020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.891643 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:35.904939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.889826 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:52:35.904939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.902153 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:52:35.904939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.902200 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:52:35.911332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.910548 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:52:35.911332 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.911115 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:35.923421 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.923390 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n"] Nov 02 21:52:35.924492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924468 4441 cpu_manager.go:414] "containerMap: removing container" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" Nov 02 21:52:35.924606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924494 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" Nov 02 21:52:35.924606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924518 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" Nov 02 21:52:35.924606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924526 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" Nov 02 21:52:35.924606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924539 4441 cpu_manager.go:414] "containerMap: removing container" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" Nov 02 21:52:35.924606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924546 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" Nov 02 21:52:35.924606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924586 4441 cpu_manager.go:414] "containerMap: removing container" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" Nov 02 21:52:35.924606 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924594 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" Nov 02 21:52:35.930422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924794 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="74df9aca-99d0-457b-b866-9e0ca409b0b3" containerName="istio-proxy" Nov 02 21:52:35.930422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924809 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="16840db4-8c59-4fbf-87a6-24a83d2b379b" containerName="istio-proxy" Nov 02 21:52:35.930422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924819 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6604492-7b11-491c-87b9-dc752ba35206" containerName="istio-proxy" Nov 02 21:52:35.930422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.924830 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="26245fbe-359b-4ff6-b6b0-58d816f553a1" containerName="istio-proxy" Nov 02 21:52:35.972636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.972602 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:52:35.972897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.972881 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:52:35.987093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.973027 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n"] Nov 02 21:52:35.987347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:35.972753 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.014433 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.013791 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:52:36.033064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.023878 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:52:36.033064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.027501 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:52:36.039650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.039612 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:52:36.060728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.043589 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:52:36.060728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.043640 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:52:36.060728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.047723 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-70d237-openshift-default-dockercfg-5x4cs\"" Nov 02 21:52:36.063572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.063484 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:36.207394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.207311 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-envoy\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.207547 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.207427 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-token\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.207547 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.207460 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-podinfo\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.208230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.208205 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-data\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.208451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.208428 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-certs\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.208543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.208503 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c41f4c-73ee-440d-befc-9def36a8b0ad-istiod-ca-cert\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.208697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.208677 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-credential-socket\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.208846 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.208825 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-socket\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.209052 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.209026 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5q4b\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-kube-api-access-q5q4b\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.309897 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.309863 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5q4b\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-kube-api-access-q5q4b\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.309930 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-envoy\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.309994 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-token\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.310015 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-podinfo\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.310070 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-data\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.310091 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-certs\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.310128 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c41f4c-73ee-440d-befc-9def36a8b0ad-istiod-ca-cert\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.310153 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-credential-socket\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310465 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.310193 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-socket\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.310634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.310578 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-socket\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.311259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.311233 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-data\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.311449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.311330 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-credential-socket\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.311633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.311605 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c41f4c-73ee-440d-befc-9def36a8b0ad-istiod-ca-cert\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.311725 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.311646 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-certs\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.313219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.313187 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-envoy\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.313317 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.313300 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-podinfo\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.389990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.383718 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-token\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.398780 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.398743 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5q4b\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-kube-api-access-q5q4b\") pod \"e2e-inf-b-70d237-openshift-default-86895f855-4l98n\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.419198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.419168 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:36.885743 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:52:36.885709 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c41f4c_73ee_440d_befc_9def36a8b0ad.slice/crio-24ead6eb556b44bd3b0308a5fdf69db7ab9f227ab1322ccec92197790f69e5a3 WatchSource:0}: Error finding container 24ead6eb556b44bd3b0308a5fdf69db7ab9f227ab1322ccec92197790f69e5a3: Status 404 returned error can't find the container with id 24ead6eb556b44bd3b0308a5fdf69db7ab9f227ab1322ccec92197790f69e5a3 Nov 02 21:52:36.887405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.887373 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n"] Nov 02 21:52:36.888885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.888846 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:36.889019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.888933 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:36.889019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:36.889001 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:37.086009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.084257 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" event={"ID":"40c41f4c-73ee-440d-befc-9def36a8b0ad","Type":"ContainerStarted","Data":"5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655"} Nov 02 21:52:37.086009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.084319 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" event={"ID":"40c41f4c-73ee-440d-befc-9def36a8b0ad","Type":"ContainerStarted","Data":"24ead6eb556b44bd3b0308a5fdf69db7ab9f227ab1322ccec92197790f69e5a3"} Nov 02 21:52:37.268691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.260701 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw295 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:52:37.268691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.260760 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:52:37.419485 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.419450 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:37.421162 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.421132 4441 patch_prober.go:29] interesting pod/e2e-inf-b-70d237-openshift-default-86895f855-4l98n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 21:52:37.421299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.421190 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 21:52:37.633982 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:37.633913 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podStartSLOduration=2.633893551 podStartE2EDuration="2.633893551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:37.626784537 +0000 UTC m=+2387.182412900" watchObservedRunningTime="2026-11-02 21:52:37.633893551 +0000 UTC m=+2387.189521889" Nov 02 21:52:38.424400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:38.424366 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:39.082387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:39.082349 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:39.103884 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:39.103844 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:39.104924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:39.104898 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:52:40.355443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.355406 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:40.526641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.526493 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:40.628020 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:40.626953 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:52:40.654932 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.654896 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:40.693756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.693733 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:52:40.797671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797639 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-data\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.797867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797695 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-credential-socket\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.797867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797761 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-socket\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.797867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797796 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-token\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-token\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.797867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797836 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc99d8fb-274c-434d-8918-235d56fb20b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc99d8fb-274c-434d-8918-235d56fb20b7-istiod-ca-cert\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.798141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797910 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-envoy\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.798141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797936 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-data" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:40.798141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797953 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-certs\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.798141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.797982 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-credential-socket" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:40.798141 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798123 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-socket" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:40.798409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798155 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-podinfo\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.798409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798191 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-kube-api-access-4pzg9\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-kube-api-access-4pzg9\") pod \"bc99d8fb-274c-434d-8918-235d56fb20b7\" (UID: \"bc99d8fb-274c-434d-8918-235d56fb20b7\") " Nov 02 21:52:40.798409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798236 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc99d8fb-274c-434d-8918-235d56fb20b7-istiod-ca-cert" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:40.798409 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798323 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-certs" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:40.798742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798719 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc99d8fb-274c-434d-8918-235d56fb20b7-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.798833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798747 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.798833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798760 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.798833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798774 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.798833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.798788 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.800386 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.800364 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-podinfo" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:40.800866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.800852 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-envoy" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:40.800960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.800947 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-token" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:40.801045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.801007 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-kube-api-access-4pzg9" pod "bc99d8fb-274c-434d-8918-235d56fb20b7" (UID: "bc99d8fb-274c-434d-8918-235d56fb20b7"). InnerVolumeSpecName "kube-api-access-4pzg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:40.900678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.900605 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.900678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.900641 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzg9\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-kube-api-access-4pzg9\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.900678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.900657 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:40.900984 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:40.900823 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc99d8fb-274c-434d-8918-235d56fb20b7-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.175155 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.175072 4441 generic.go:356] "Generic (PLEG): container finished" podID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerID="7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071" exitCode=0 Nov 02 21:52:41.175524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.175496 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" event={"ID":"bc99d8fb-274c-434d-8918-235d56fb20b7","Type":"ContainerDied","Data":"7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071"} Nov 02 21:52:41.175704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.175687 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" event={"ID":"bc99d8fb-274c-434d-8918-235d56fb20b7","Type":"ContainerDied","Data":"590ba63ced125a14ea99a43146f2ced7014e41ba6405ecb2fbaea3967909718a"} Nov 02 21:52:41.176475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.175874 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295" Nov 02 21:52:41.185164 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.175889 4441 scope.go:122] "RemoveContainer" containerID="7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071" Nov 02 21:52:41.191259 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.191227 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:52:41.209991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.209925 4441 generic.go:356] "Generic (PLEG): container finished" podID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerID="fcb0121ff601e6db2db900403079d5b5d8e9d1aaafbe1af345370e14b84212e6" exitCode=0 Nov 02 21:52:41.210113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.210060 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" event={"ID":"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09","Type":"ContainerDied","Data":"fcb0121ff601e6db2db900403079d5b5d8e9d1aaafbe1af345370e14b84212e6"} Nov 02 21:52:41.215001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.212617 4441 generic.go:356] "Generic (PLEG): container finished" podID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerID="5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e" exitCode=0 Nov 02 21:52:41.215001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.212758 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" event={"ID":"7ce53479-cad7-4031-b32f-3b1ba1d5eed2","Type":"ContainerDied","Data":"5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e"} Nov 02 21:52:41.215001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.212795 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" event={"ID":"7ce53479-cad7-4031-b32f-3b1ba1d5eed2","Type":"ContainerDied","Data":"c17d4078cdb74c439a19922f3885d0c89ae2f9598bfc615d8548a3355c355936"} Nov 02 21:52:41.215001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.212872 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2" Nov 02 21:52:41.215001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.214836 4441 generic.go:356] "Generic (PLEG): container finished" podID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerID="7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16" exitCode=0 Nov 02 21:52:41.216238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.215390 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" event={"ID":"f2719ffe-8af4-4118-bf17-be831cfb32da","Type":"ContainerDied","Data":"7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16"} Nov 02 21:52:41.216238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.215784 4441 scope.go:122] "RemoveContainer" containerID="7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16" Nov 02 21:52:41.234986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.230490 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:41.234986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.230584 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:41.234986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.230637 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:41.240104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.239421 4441 scope.go:122] "RemoveContainer" containerID="7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071" Nov 02 21:52:41.243986 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:41.241432 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071\": container with ID starting with 7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071 not found: ID does not exist" containerID="7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071" Nov 02 21:52:41.243986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.241470 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071"} err="failed to get container status \"7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071\": rpc error: code = NotFound desc = could not find container \"7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071\": container with ID starting with 7bd4350399ec35dfe1d69ddb017f936180423bb39b9f0c1b3a00b0f79a412071 not found: ID does not exist" Nov 02 21:52:41.243986 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.241495 4441 scope.go:122] "RemoveContainer" containerID="5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e" Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.309770 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-podinfo\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.309833 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-socket\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.309867 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-token\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-token\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.309927 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istiod-ca-cert\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.309995 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-certs\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.310098 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-kube-api-access-4lnjf\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-kube-api-access-4lnjf\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.310218 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-envoy\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.310290 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-credential-socket\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.311928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.310382 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-data\") pod \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\" (UID: \"7ce53479-cad7-4031-b32f-3b1ba1d5eed2\") " Nov 02 21:52:41.312669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.312220 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-data" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:41.312669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.312412 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-socket" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:41.328243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.327622 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-certs" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:41.330024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.328492 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-token" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:41.333992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.333298 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-credential-socket" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:41.339702 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.339676 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istiod-ca-cert" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:41.405989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.396393 4441 scope.go:122] "RemoveContainer" containerID="5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e" Nov 02 21:52:41.408039 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:52:41.408001 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e\": container with ID starting with 5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e not found: ID does not exist" containerID="5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e" Nov 02 21:52:41.408189 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.408070 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e"} err="failed to get container status \"5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e\": rpc error: code = NotFound desc = could not find container \"5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e\": container with ID starting with 5d001f0579e6e8eeebc69775f58e1ad65f2776a7e893e678bda6c1eacedff21e not found: ID does not exist" Nov 02 21:52:41.431843 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.431755 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.432122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.432101 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.432282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.432267 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.432381 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.432370 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.432491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.432478 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.432605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.432592 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.595101 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.595036 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-podinfo" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:41.597828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.597787 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-envoy" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:41.707207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.695752 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.707207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.695793 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.737989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.735747 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295"] Nov 02 21:52:41.751463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.751427 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-kube-api-access-4lnjf" pod "7ce53479-cad7-4031-b32f-3b1ba1d5eed2" (UID: "7ce53479-cad7-4031-b32f-3b1ba1d5eed2"). InnerVolumeSpecName "kube-api-access-4lnjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:41.758999 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.758958 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw295"] Nov 02 21:52:41.799506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.797004 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lnjf\" (UniqueName: \"kubernetes.io/projected/7ce53479-cad7-4031-b32f-3b1ba1d5eed2-kube-api-access-4lnjf\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:41.924774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.924095 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:52:41.950037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.950006 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2"] Nov 02 21:52:41.956658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.956612 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6swr2"] Nov 02 21:52:41.999472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999396 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-socket\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:41.999637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999491 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istiod-ca-cert\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:41.999637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999529 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-data\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:41.999637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999586 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-token\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-token\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:41.999822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999643 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-socket" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:41.999822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999684 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-envoy\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:41.999822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999712 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-podinfo\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:41.999822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999759 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-kube-api-access-kz8n8\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-kube-api-access-kz8n8\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:41.999822 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999792 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-certs\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:42.000114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999873 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-data" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:42.000114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999876 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-credential-socket\") pod \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\" (UID: \"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09\") " Nov 02 21:52:42.000114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:41.999925 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istiod-ca-cert" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:42.000294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.000119 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-credential-socket" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:42.000357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.000300 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-certs" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:42.000629 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.000605 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.000732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.000636 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.000732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.000652 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.000732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.000667 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.000732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.000681 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.002502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.002456 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-token" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:42.002649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.002622 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-kube-api-access-kz8n8" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "kube-api-access-kz8n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:42.002649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.002639 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-podinfo" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:42.003610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.003583 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-envoy" pod "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" (UID: "da4b83ca-ee48-4d75-aaa4-d58a5f23eb09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:42.102576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.102480 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.102576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.102521 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.102576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.102534 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.102576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.102548 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8n8\" (UniqueName: \"kubernetes.io/projected/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09-kube-api-access-kz8n8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:42.239156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.239113 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" event={"ID":"f2719ffe-8af4-4118-bf17-be831cfb32da","Type":"ContainerStarted","Data":"ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe"} Nov 02 21:52:42.243759 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.243734 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" Nov 02 21:52:42.243998 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.243768 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" event={"ID":"da4b83ca-ee48-4d75-aaa4-d58a5f23eb09","Type":"ContainerDied","Data":"42e706865a3ea1c246153fdafe374f449e672c265808df40a6c24cea8f96bc5d"} Nov 02 21:52:42.244137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.244035 4441 scope.go:122] "RemoveContainer" containerID="fcb0121ff601e6db2db900403079d5b5d8e9d1aaafbe1af345370e14b84212e6" Nov 02 21:52:42.326988 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.326933 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:52:42.328177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.328145 4441 patch_prober.go:29] interesting pod/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 21:52:42.328347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.328202 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 21:52:42.504658 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.504614 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928"] Nov 02 21:52:42.509022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.508937 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928"] Nov 02 21:52:42.533500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.533472 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:52:42.588515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.588482 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" path="/var/lib/kubelet/pods/7ce53479-cad7-4031-b32f-3b1ba1d5eed2/volumes" Nov 02 21:52:42.589374 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.589341 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" path="/var/lib/kubelet/pods/bc99d8fb-274c-434d-8918-235d56fb20b7/volumes" Nov 02 21:52:42.589992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.589957 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" path="/var/lib/kubelet/pods/da4b83ca-ee48-4d75-aaa4-d58a5f23eb09/volumes" Nov 02 21:52:42.911728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.911687 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cd928 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:52:42.911976 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:42.911766 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cd928" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:52:43.327657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:43.327619 4441 patch_prober.go:29] interesting pod/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 21:52:43.327816 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:43.327682 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 21:52:43.593414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:43.592984 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:43.839534 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:43.833950 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:44.012135 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.011590 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg"] Nov 02 21:52:44.012392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012370 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" Nov 02 21:52:44.012467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012392 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" Nov 02 21:52:44.012467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012406 4441 cpu_manager.go:414] "containerMap: removing container" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" Nov 02 21:52:44.012467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012411 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" Nov 02 21:52:44.012467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012449 4441 cpu_manager.go:414] "containerMap: removing container" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" Nov 02 21:52:44.012467 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012454 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" Nov 02 21:52:44.012731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012543 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc99d8fb-274c-434d-8918-235d56fb20b7" containerName="istio-proxy" Nov 02 21:52:44.012731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012555 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce53479-cad7-4031-b32f-3b1ba1d5eed2" containerName="istio-proxy" Nov 02 21:52:44.012731 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.012569 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4b83ca-ee48-4d75-aaa4-d58a5f23eb09" containerName="istio-proxy" Nov 02 21:52:44.017835 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.017812 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.114699 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.114661 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg"] Nov 02 21:52:44.115266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.115241 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-70d237-dockercfg-dzppz\"" Nov 02 21:52:44.118990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.115460 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-70d237\"" Nov 02 21:52:44.211428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.211383 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.211816 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.211788 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkd6w\" (UniqueName: \"kubernetes.io/projected/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-kube-api-access-rkd6w\") pod \"payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.243495 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.243456 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:44.254578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.254547 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72"] Nov 02 21:52:44.259251 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.259223 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:44.324071 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.270845 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72"] Nov 02 21:52:44.333425 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.333378 4441 patch_prober.go:29] interesting pod/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 21:52:44.333685 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.333657 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 21:52:44.353926 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.345350 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.354120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.346207 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.354207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.354117 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkd6w\" (UniqueName: \"kubernetes.io/projected/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-kube-api-access-rkd6w\") pod \"payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.412006 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.411069 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkd6w\" (UniqueName: \"kubernetes.io/projected/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-kube-api-access-rkd6w\") pod \"payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.456009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.455771 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wz7\" (UniqueName: \"kubernetes.io/projected/e17dd125-18dd-4446-8dbd-887e19cd0461-kube-api-access-j4wz7\") pod \"payload-processing-e2e-inf-b-70d237-c848cd89-hph72\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:44.456009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.455872 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17dd125-18dd-4446-8dbd-887e19cd0461-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70d237-c848cd89-hph72\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:44.556888 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.556843 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wz7\" (UniqueName: \"kubernetes.io/projected/e17dd125-18dd-4446-8dbd-887e19cd0461-kube-api-access-j4wz7\") pod \"payload-processing-e2e-inf-b-70d237-c848cd89-hph72\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:44.557087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.556940 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17dd125-18dd-4446-8dbd-887e19cd0461-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70d237-c848cd89-hph72\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:44.557712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.557682 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17dd125-18dd-4446-8dbd-887e19cd0461-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70d237-c848cd89-hph72\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:44.612936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.612857 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wz7\" (UniqueName: \"kubernetes.io/projected/e17dd125-18dd-4446-8dbd-887e19cd0461-kube-api-access-j4wz7\") pod \"payload-processing-e2e-inf-b-70d237-c848cd89-hph72\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:44.634449 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.634422 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:44.667149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:44.667121 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:45.042390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.042359 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg"] Nov 02 21:52:45.042907 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:52:45.042879 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaaff1d_b8c5_44b4_9fa5_f44e795575f8.slice/crio-a50811470d06c9d77519821a9cc235af13354788ee92fbfcb0f8272f1c66d9f0 WatchSource:0}: Error finding container a50811470d06c9d77519821a9cc235af13354788ee92fbfcb0f8272f1c66d9f0: Status 404 returned error can't find the container with id a50811470d06c9d77519821a9cc235af13354788ee92fbfcb0f8272f1c66d9f0 Nov 02 21:52:45.140111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.140085 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72"] Nov 02 21:52:45.141096 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:52:45.141072 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17dd125_18dd_4446_8dbd_887e19cd0461.slice/crio-348014309d21badbb8e8195cee2be0bd86a0a4f264bd5d92c7bdd7ea549db238 WatchSource:0}: Error finding container 348014309d21badbb8e8195cee2be0bd86a0a4f264bd5d92c7bdd7ea549db238: Status 404 returned error can't find the container with id 348014309d21badbb8e8195cee2be0bd86a0a4f264bd5d92c7bdd7ea549db238 Nov 02 21:52:45.327316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.327282 4441 patch_prober.go:29] interesting pod/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 21:52:45.327484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.327336 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 21:52:45.367086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.367051 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" event={"ID":"e17dd125-18dd-4446-8dbd-887e19cd0461","Type":"ContainerStarted","Data":"6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4"} Nov 02 21:52:45.367253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.367110 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" event={"ID":"e17dd125-18dd-4446-8dbd-887e19cd0461","Type":"ContainerStarted","Data":"348014309d21badbb8e8195cee2be0bd86a0a4f264bd5d92c7bdd7ea549db238"} Nov 02 21:52:45.367320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.367252 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:45.368867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.368838 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" event={"ID":"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8","Type":"ContainerStarted","Data":"91f625f88d3c10c6d1127440a7e527e53a78023a5ee715ae578c25fc8d7b3a81"} Nov 02 21:52:45.369022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.368889 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" event={"ID":"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8","Type":"ContainerStarted","Data":"a50811470d06c9d77519821a9cc235af13354788ee92fbfcb0f8272f1c66d9f0"} Nov 02 21:52:45.369022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.368922 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:45.499992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.499366 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" podStartSLOduration=1.4993471299999999 podStartE2EDuration="1.49934713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:45.498191595 +0000 UTC m=+2395.053819961" watchObservedRunningTime="2026-11-02 21:52:45.49934713 +0000 UTC m=+2395.054975469" Nov 02 21:52:45.615047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:45.610833 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" podStartSLOduration=2.610811612 podStartE2EDuration="2.610811612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:45.607400234 +0000 UTC m=+2395.163028571" watchObservedRunningTime="2026-11-02 21:52:45.610811612 +0000 UTC m=+2395.166439949" Nov 02 21:52:46.327781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:46.327741 4441 patch_prober.go:29] interesting pod/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 21:52:46.328029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:46.327803 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 21:52:47.327102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:47.327061 4441 patch_prober.go:29] interesting pod/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 21:52:47.327504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:47.327119 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 21:52:48.331358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:48.331327 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:52:48.332366 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:48.332346 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 21:52:49.010306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.010269 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:49.432044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.432011 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj"] Nov 02 21:52:49.436327 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.436310 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.446207 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.446189 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr"] Nov 02 21:52:49.449960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.449942 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g"] Nov 02 21:52:49.450064 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.450046 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.454066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.454050 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.530223 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.530195 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj"] Nov 02 21:52:49.554450 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554421 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.554634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554456 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.554634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554494 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spnr\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-kube-api-access-7spnr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.554634 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554616 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.554814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554661 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.554814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554677 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.554814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554700 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.554814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554721 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.554814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554741 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.554814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554765 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d28baed-dc19-4f33-803a-8a19e68d50c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.554814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554811 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554841 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554868 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0b4b6-6923-42f2-8595-6855d32a0053-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554887 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchjz\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-kube-api-access-lchjz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554937 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.554962 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555026 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555041 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555062 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555080 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555097 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555110 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555131 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4t6\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-kube-api-access-pm4t6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555154 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.555170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555170 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.555639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555190 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.555639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.555211 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.579445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.579409 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g"] Nov 02 21:52:49.597537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.597511 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr"] Nov 02 21:52:49.656257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656225 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0b4b6-6923-42f2-8595-6855d32a0053-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.656499 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656476 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchjz\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-kube-api-access-lchjz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.656691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656671 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.656839 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656817 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.656930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656890 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.656930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656916 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.657063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656950 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.656998 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.657063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657024 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657063 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657048 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657083 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4t6\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-kube-api-access-pm4t6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657116 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657141 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657174 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657210 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657227 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657233 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.657278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657261 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657291 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0b4b6-6923-42f2-8595-6855d32a0053-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.657691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657334 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.657691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657377 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.657691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657419 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657481 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spnr\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-kube-api-access-7spnr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.657691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657488 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.657691 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657611 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657718 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657790 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657857 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657889 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657932 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657962 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.657985 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658041 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.658074 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658060 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.658578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658083 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d28baed-dc19-4f33-803a-8a19e68d50c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.658578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658155 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.658578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658207 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.658578 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658335 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.658812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658800 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.659019 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.658959 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d28baed-dc19-4f33-803a-8a19e68d50c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.659171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.659058 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.659560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.659538 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.660809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.660018 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.660924 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.660851 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.661024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.660961 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.661335 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.661308 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.661428 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.661412 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.661500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.661481 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.711778 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.711720 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.712307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.711998 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchjz\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-kube-api-access-lchjz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.712683 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.712653 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.712831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.712745 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spnr\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-kube-api-access-7spnr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kch8g\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:49.713883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.713853 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s5kwr\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.714320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.714303 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4t6\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-kube-api-access-pm4t6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-662qj\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.751546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.751521 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:49.763912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.763599 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:49.771174 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:49.771150 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:50.170794 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:52:50.170758 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e7d807_33c6_49a7_801a_1e1be51c7eab.slice/crio-3d59ebd75cfac3c7f56960049494a6c0a63f47d771d03d93e496d14147af8474 WatchSource:0}: Error finding container 3d59ebd75cfac3c7f56960049494a6c0a63f47d771d03d93e496d14147af8474: Status 404 returned error can't find the container with id 3d59ebd75cfac3c7f56960049494a6c0a63f47d771d03d93e496d14147af8474 Nov 02 21:52:50.171285 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.171261 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj"] Nov 02 21:52:50.173690 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.173657 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.173823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.173740 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.173823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.173782 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.182451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.182428 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr"] Nov 02 21:52:50.186182 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:52:50.186159 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e0b4b6_6923_42f2_8595_6855d32a0053.slice/crio-bbe2e8f50385c5f9b058d21de969919faff5845c7d4a66ec806c44f60c49b70c WatchSource:0}: Error finding container bbe2e8f50385c5f9b058d21de969919faff5845c7d4a66ec806c44f60c49b70c: Status 404 returned error can't find the container with id bbe2e8f50385c5f9b058d21de969919faff5845c7d4a66ec806c44f60c49b70c Nov 02 21:52:50.189013 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.188981 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.189127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.189058 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.189127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.189096 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.208407 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.208381 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g"] Nov 02 21:52:50.217390 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:52:50.217364 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d28baed_dc19_4f33_803a_8a19e68d50c0.slice/crio-5aee4b0149f02b0b4041f85498dec5469ecff25cc86e3e1d02044db0f541471d WatchSource:0}: Error finding container 5aee4b0149f02b0b4041f85498dec5469ecff25cc86e3e1d02044db0f541471d: Status 404 returned error can't find the container with id 5aee4b0149f02b0b4041f85498dec5469ecff25cc86e3e1d02044db0f541471d Nov 02 21:52:50.220297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.220119 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.220297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.220185 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.220297 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.220225 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:52:50.442901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.442819 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" event={"ID":"d4e7d807-33c6-49a7-801a-1e1be51c7eab","Type":"ContainerStarted","Data":"889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48"} Nov 02 21:52:50.443452 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.443425 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" event={"ID":"d4e7d807-33c6-49a7-801a-1e1be51c7eab","Type":"ContainerStarted","Data":"3d59ebd75cfac3c7f56960049494a6c0a63f47d771d03d93e496d14147af8474"} Nov 02 21:52:50.445165 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.445133 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" event={"ID":"5d28baed-dc19-4f33-803a-8a19e68d50c0","Type":"ContainerStarted","Data":"138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9"} Nov 02 21:52:50.445305 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.445188 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" event={"ID":"5d28baed-dc19-4f33-803a-8a19e68d50c0","Type":"ContainerStarted","Data":"5aee4b0149f02b0b4041f85498dec5469ecff25cc86e3e1d02044db0f541471d"} Nov 02 21:52:50.447355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.447331 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" event={"ID":"d4e0b4b6-6923-42f2-8595-6855d32a0053","Type":"ContainerStarted","Data":"4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee"} Nov 02 21:52:50.447466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.447369 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" event={"ID":"d4e0b4b6-6923-42f2-8595-6855d32a0053","Type":"ContainerStarted","Data":"bbe2e8f50385c5f9b058d21de969919faff5845c7d4a66ec806c44f60c49b70c"} Nov 02 21:52:50.556615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.556546 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podStartSLOduration=1.556510631 podStartE2EDuration="1.556510631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:50.552898011 +0000 UTC m=+2400.108526412" watchObservedRunningTime="2026-11-02 21:52:50.556510631 +0000 UTC m=+2400.112138970" Nov 02 21:52:50.670890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.670836 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podStartSLOduration=1.6708176959999999 podStartE2EDuration="1.670817696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:50.662520514 +0000 UTC m=+2400.218148853" watchObservedRunningTime="2026-11-02 21:52:50.670817696 +0000 UTC m=+2400.226446034" Nov 02 21:52:50.752170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.752094 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:50.765067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.765031 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:50.771451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.771366 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:50.776498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:50.776451 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podStartSLOduration=1.776438247 podStartE2EDuration="1.776438247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:50.771793047 +0000 UTC m=+2400.327421386" watchObservedRunningTime="2026-11-02 21:52:50.776438247 +0000 UTC m=+2400.332066583" Nov 02 21:52:51.401127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.401096 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:52:51.422324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.422294 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:52:51.466037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.465995 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:52:51.468546 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.468520 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:52:51.555152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.555119 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:52:51.559907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.559889 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:52:51.754046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.753919 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-662qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 21:52:51.754046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.754017 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 21:52:51.765956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.765924 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 21:52:51.766104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.765979 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 21:52:51.774990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.774602 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kch8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 02 21:52:51.774990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:51.774650 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 02 21:52:52.752418 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:52.752375 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-662qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 21:52:52.752910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:52.752430 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 21:52:52.764735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:52.764701 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 21:52:52.764878 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:52.764749 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 21:52:52.784990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:52.783279 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kch8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 02 21:52:52.784990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:52.783335 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 02 21:52:53.550997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.547911 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:53.752844 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.752789 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-662qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 21:52:53.753265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.752861 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 21:52:53.764861 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.764838 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 21:52:53.764936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.764888 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 21:52:53.784992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.781349 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kch8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 02 21:52:53.784992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.781402 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 02 21:52:53.807042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.798047 4441 scope.go:122] "RemoveContainer" containerID="c25c35a70700f7d63a3ec1d9712319947e8ca22b70ad27165ebffa17f59f56cc" Nov 02 21:52:53.850565 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.850537 4441 scope.go:122] "RemoveContainer" containerID="6c30da7ad5df625d272a0cf76792887dc4591a808fd75d759371b253b236c8cb" Nov 02 21:52:53.861232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.861213 4441 scope.go:122] "RemoveContainer" containerID="880c86fe2c0eda6dce50a6ef76ac5f829dcade0cb45a416c089e3089d5538cbe" Nov 02 21:52:53.893876 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.892133 4441 scope.go:122] "RemoveContainer" containerID="e850903cfc42532153587a5d13b325798a749d3044aab04b795a6e09e241e8ce" Nov 02 21:52:53.956316 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:53.956171 4441 scope.go:122] "RemoveContainer" containerID="fc41c32d45bb50a3352af984a0f712e04ab5ef5848602bb46a553ed86100cdcc" Nov 02 21:52:54.752515 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:54.752477 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-662qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 21:52:54.752698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:54.752535 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 21:52:54.764810 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:54.764786 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 21:52:54.765138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:54.764832 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 21:52:54.790992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:54.790073 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kch8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 02 21:52:54.790992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:54.790124 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 02 21:52:54.892991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:54.892477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:55.752445 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:55.752398 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-662qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 21:52:55.752677 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:55.752463 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 21:52:55.764949 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:55.764916 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 21:52:55.765311 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:55.764957 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 21:52:55.776997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:55.773225 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kch8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 02 21:52:55.776997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:55.773269 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 02 21:52:56.444994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.444131 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 21:52:56.445469 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.445447 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 21:52:56.751964 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.751855 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-662qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 21:52:56.752282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.752243 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 21:52:56.764769 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.764735 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 21:52:56.764915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.764792 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 21:52:56.772479 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.772448 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kch8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 02 21:52:56.772920 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:56.772497 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 02 21:52:57.752245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:57.752203 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-662qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 21:52:57.752451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:57.752256 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 21:52:57.764569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:57.764532 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 21:52:57.764722 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:57.764607 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 21:52:57.772480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:57.772456 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kch8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body= Nov 02 21:52:57.772611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:57.772501 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" Nov 02 21:52:57.943201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:57.943166 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:59.254181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.252774 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:59.254693 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.254415 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:59.255296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.255247 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:52:59.516787 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.516697 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:59.517180 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.517122 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:59.518166 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.518142 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:52:59.687020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.686917 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:59.688537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.688513 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:52:59.689741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:52:59.689717 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:53:02.402697 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:02.402664 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:06.834487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:06.834451 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:06.980444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:06.980381 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:07.097801 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:07.097721 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:07.351464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:07.351380 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:07.475550 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:07.475518 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:10.618689 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:53:10.618647 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:53:11.796149 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:11.796107 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:12.526630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:12.526595 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:15.408714 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:15.408676 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:16.521962 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:16.521921 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:16.761245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:16.761213 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:21.096571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:21.096532 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:24.928191 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:24.928154 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:25.355903 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:25.355870 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:25.942990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:25.942746 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:26.337173 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:26.337137 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:26.599953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:26.599847 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:27.249866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.249831 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz"] Nov 02 21:53:27.256130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.256109 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.276198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.276171 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.276328 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.276240 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqplp\" (UniqueName: \"kubernetes.io/projected/07340c30-44f9-467a-a590-30b8cca7b737-kube-api-access-fqplp\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.276430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.276408 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.305614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.305584 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz"] Nov 02 21:53:27.346507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.346475 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:53:27.346507 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.346496 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:53:27.346750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.346564 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:53:27.346750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.346600 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:53:27.377471 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.377445 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.377604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.377513 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqplp\" (UniqueName: \"kubernetes.io/projected/07340c30-44f9-467a-a590-30b8cca7b737-kube-api-access-fqplp\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.377672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.377610 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.378175 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.378154 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.380156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.380134 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.431096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.431073 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqplp\" (UniqueName: \"kubernetes.io/projected/07340c30-44f9-467a-a590-30b8cca7b737-kube-api-access-fqplp\") pod \"maas-api-revoke-sub-9be55312-gtvcz\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.569617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.569586 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:27.971040 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:53:27.970995 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07340c30_44f9_467a_a590_30b8cca7b737.slice/crio-789418173d4760b90f3ca6c1cde655913ceff296b882c06bab07f8548cf7555c WatchSource:0}: Error finding container 789418173d4760b90f3ca6c1cde655913ceff296b882c06bab07f8548cf7555c: Status 404 returned error can't find the container with id 789418173d4760b90f3ca6c1cde655913ceff296b882c06bab07f8548cf7555c Nov 02 21:53:27.971262 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:27.971244 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz"] Nov 02 21:53:28.892948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:28.892921 4441 generic.go:356] "Generic (PLEG): container finished" podID="07340c30-44f9-467a-a590-30b8cca7b737" containerID="448e63af1ca2e9b354f0546cb7513fbd1ddd606035667727ddcd6d9373982824" exitCode=0 Nov 02 21:53:28.893357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:28.893011 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" event={"ID":"07340c30-44f9-467a-a590-30b8cca7b737","Type":"ContainerDied","Data":"448e63af1ca2e9b354f0546cb7513fbd1ddd606035667727ddcd6d9373982824"} Nov 02 21:53:28.893357 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:28.893070 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" event={"ID":"07340c30-44f9-467a-a590-30b8cca7b737","Type":"ContainerStarted","Data":"789418173d4760b90f3ca6c1cde655913ceff296b882c06bab07f8548cf7555c"} Nov 02 21:53:30.094056 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.094032 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:30.108007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.107985 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-openshift-ca\") pod \"07340c30-44f9-467a-a590-30b8cca7b737\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " Nov 02 21:53:30.108119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.108026 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-xks-ca\") pod \"07340c30-44f9-467a-a590-30b8cca7b737\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " Nov 02 21:53:30.108198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.108185 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07340c30-44f9-467a-a590-30b8cca7b737-kube-api-access-fqplp\" (UniqueName: \"kubernetes.io/projected/07340c30-44f9-467a-a590-30b8cca7b737-kube-api-access-fqplp\") pod \"07340c30-44f9-467a-a590-30b8cca7b737\" (UID: \"07340c30-44f9-467a-a590-30b8cca7b737\") " Nov 02 21:53:30.108346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.108321 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-openshift-ca" pod "07340c30-44f9-467a-a590-30b8cca7b737" (UID: "07340c30-44f9-467a-a590-30b8cca7b737"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:53:30.108688 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.108666 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:30.110130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.110106 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07340c30-44f9-467a-a590-30b8cca7b737-kube-api-access-fqplp" pod "07340c30-44f9-467a-a590-30b8cca7b737" (UID: "07340c30-44f9-467a-a590-30b8cca7b737"). InnerVolumeSpecName "kube-api-access-fqplp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:53:30.110212 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.110139 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-xks-ca" pod "07340c30-44f9-467a-a590-30b8cca7b737" (UID: "07340c30-44f9-467a-a590-30b8cca7b737"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:53:30.209964 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.209888 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07340c30-44f9-467a-a590-30b8cca7b737-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:30.209964 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.209927 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqplp\" (UniqueName: \"kubernetes.io/projected/07340c30-44f9-467a-a590-30b8cca7b737-kube-api-access-fqplp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:30.423820 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.423781 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:30.595146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.595115 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:30.720772 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.720738 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:30.886456 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.886374 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:30.917562 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.917528 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" event={"ID":"07340c30-44f9-467a-a590-30b8cca7b737","Type":"ContainerDied","Data":"789418173d4760b90f3ca6c1cde655913ceff296b882c06bab07f8548cf7555c"} Nov 02 21:53:30.917749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.917589 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789418173d4760b90f3ca6c1cde655913ceff296b882c06bab07f8548cf7555c" Nov 02 21:53:30.917749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:30.917534 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz" Nov 02 21:53:33.604020 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.603982 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:33.747661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.747566 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:33.943823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.943732 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz"] Nov 02 21:53:33.944365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.944349 4441 cpu_manager.go:414] "containerMap: removing container" podUID="07340c30-44f9-467a-a590-30b8cca7b737" containerName="revoke-keys" Nov 02 21:53:33.944365 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.944365 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07340c30-44f9-467a-a590-30b8cca7b737" containerName="revoke-keys" Nov 02 21:53:33.944484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.944476 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="07340c30-44f9-467a-a590-30b8cca7b737" containerName="revoke-keys" Nov 02 21:53:33.948001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.947979 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:33.968298 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.968275 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxjb\" (UniqueName: \"kubernetes.io/projected/b3f7af50-2c16-4253-9666-9b20fcc311b7-kube-api-access-skxjb\") pod \"payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:33.968454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:33.968433 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f7af50-2c16-4253-9666-9b20fcc311b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:34.001785 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.001762 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz"] Nov 02 21:53:34.047442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.047231 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bcb857\"" Nov 02 21:53:34.047442 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.047407 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bcb857-dockercfg-zxnd6\"" Nov 02 21:53:34.088947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.088903 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f7af50-2c16-4253-9666-9b20fcc311b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:34.095293 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.095262 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxjb\" (UniqueName: \"kubernetes.io/projected/b3f7af50-2c16-4253-9666-9b20fcc311b7-kube-api-access-skxjb\") pod \"payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:34.103238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.095747 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f7af50-2c16-4253-9666-9b20fcc311b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:34.166070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.166033 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxjb\" (UniqueName: \"kubernetes.io/projected/b3f7af50-2c16-4253-9666-9b20fcc311b7-kube-api-access-skxjb\") pod \"payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:34.261803 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.261719 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:34.651351 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:53:34.651314 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f7af50_2c16_4253_9666_9b20fcc311b7.slice/crio-a68280d3fb5356f68032c946bfd94d6ab42bbed6d7fe08861c7014e5943c669a WatchSource:0}: Error finding container a68280d3fb5356f68032c946bfd94d6ab42bbed6d7fe08861c7014e5943c669a: Status 404 returned error can't find the container with id a68280d3fb5356f68032c946bfd94d6ab42bbed6d7fe08861c7014e5943c669a Nov 02 21:53:34.651877 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.651853 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz"] Nov 02 21:53:34.948831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.948734 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" event={"ID":"b3f7af50-2c16-4253-9666-9b20fcc311b7","Type":"ContainerStarted","Data":"f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410"} Nov 02 21:53:34.948831 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.948792 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" event={"ID":"b3f7af50-2c16-4253-9666-9b20fcc311b7","Type":"ContainerStarted","Data":"a68280d3fb5356f68032c946bfd94d6ab42bbed6d7fe08861c7014e5943c669a"} Nov 02 21:53:34.949062 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:34.948929 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:35.057244 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:35.057184 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" podStartSLOduration=2.057170865 podStartE2EDuration="2.057170865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:35.055555965 +0000 UTC m=+2444.611184313" watchObservedRunningTime="2026-11-02 21:53:35.057170865 +0000 UTC m=+2444.612799244" Nov 02 21:53:35.832799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:35.832754 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:35.981608 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:35.981576 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:36.157046 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:36.155418 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:37.915282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:37.915242 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:38.390116 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.390085 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb"] Nov 02 21:53:38.394430 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.394404 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.445823 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.445786 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb"] Nov 02 21:53:38.467712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.467685 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.467855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.467746 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlhh\" (UniqueName: \"kubernetes.io/projected/5d394cb1-c26f-4351-8d50-fa991b38e8f1-kube-api-access-6mlhh\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.467855 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.467815 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.488636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.488606 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:53:38.488856 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.488395 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:53:38.489018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.488808 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:53:38.490912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.490882 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:53:38.569291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.569262 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.569459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.569322 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlhh\" (UniqueName: \"kubernetes.io/projected/5d394cb1-c26f-4351-8d50-fa991b38e8f1-kube-api-access-6mlhh\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.569459 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.569351 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.570038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.570015 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.571864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.571830 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.623030 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.623001 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlhh\" (UniqueName: \"kubernetes.io/projected/5d394cb1-c26f-4351-8d50-fa991b38e8f1-kube-api-access-6mlhh\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:38.716270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:38.716179 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:39.161709 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:53:39.161680 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d394cb1_c26f_4351_8d50_fa991b38e8f1.slice/crio-4ba70f571931f429bac426bf194356600f9548332db91040f853895891bc41cd WatchSource:0}: Error finding container 4ba70f571931f429bac426bf194356600f9548332db91040f853895891bc41cd: Status 404 returned error can't find the container with id 4ba70f571931f429bac426bf194356600f9548332db91040f853895891bc41cd Nov 02 21:53:39.162703 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:39.162681 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb"] Nov 02 21:53:40.001790 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:40.001752 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" event={"ID":"5d394cb1-c26f-4351-8d50-fa991b38e8f1","Type":"ContainerStarted","Data":"612477f45b5d7cfcc32275ec1166ca34ee2bbb120fff15eafdc4def5ef53744e"} Nov 02 21:53:40.002033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:40.001807 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" event={"ID":"5d394cb1-c26f-4351-8d50-fa991b38e8f1","Type":"ContainerStarted","Data":"4ba70f571931f429bac426bf194356600f9548332db91040f853895891bc41cd"} Nov 02 21:53:40.111809 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:40.111743 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" podStartSLOduration=2.111725595 podStartE2EDuration="2.111725595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:40.108659121 +0000 UTC m=+2449.664287459" watchObservedRunningTime="2026-11-02 21:53:40.111725595 +0000 UTC m=+2449.667353934" Nov 02 21:53:40.618928 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:53:40.618891 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:53:40.764558 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:40.764517 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:41.012636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:41.012544 4441 generic.go:356] "Generic (PLEG): container finished" podID="5d394cb1-c26f-4351-8d50-fa991b38e8f1" containerID="612477f45b5d7cfcc32275ec1166ca34ee2bbb120fff15eafdc4def5ef53744e" exitCode=0 Nov 02 21:53:41.012789 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:41.012627 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" event={"ID":"5d394cb1-c26f-4351-8d50-fa991b38e8f1","Type":"ContainerDied","Data":"612477f45b5d7cfcc32275ec1166ca34ee2bbb120fff15eafdc4def5ef53744e"} Nov 02 21:53:42.258422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.258395 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:42.283542 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.283506 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d394cb1-c26f-4351-8d50-fa991b38e8f1-kube-api-access-6mlhh\" (UniqueName: \"kubernetes.io/projected/5d394cb1-c26f-4351-8d50-fa991b38e8f1-kube-api-access-6mlhh\") pod \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " Nov 02 21:53:42.284102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.284071 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-openshift-ca\") pod \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " Nov 02 21:53:42.284245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.284171 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-xks-ca\") pod \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\" (UID: \"5d394cb1-c26f-4351-8d50-fa991b38e8f1\") " Nov 02 21:53:42.284596 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.284570 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-openshift-ca" pod "5d394cb1-c26f-4351-8d50-fa991b38e8f1" (UID: "5d394cb1-c26f-4351-8d50-fa991b38e8f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:53:42.284867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.284840 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:42.286093 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.286066 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d394cb1-c26f-4351-8d50-fa991b38e8f1-kube-api-access-6mlhh" pod "5d394cb1-c26f-4351-8d50-fa991b38e8f1" (UID: "5d394cb1-c26f-4351-8d50-fa991b38e8f1"). InnerVolumeSpecName "kube-api-access-6mlhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:53:42.286687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.286661 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-xks-ca" pod "5d394cb1-c26f-4351-8d50-fa991b38e8f1" (UID: "5d394cb1-c26f-4351-8d50-fa991b38e8f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:53:42.386253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.386164 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d394cb1-c26f-4351-8d50-fa991b38e8f1-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:42.386253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.386206 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mlhh\" (UniqueName: \"kubernetes.io/projected/5d394cb1-c26f-4351-8d50-fa991b38e8f1-kube-api-access-6mlhh\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:42.888657 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.888619 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6"] Nov 02 21:53:42.916586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:42.916551 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v"] Nov 02 21:53:43.083767 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:43.083713 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:43.093388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:43.093345 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" event={"ID":"5d394cb1-c26f-4351-8d50-fa991b38e8f1","Type":"ContainerDied","Data":"4ba70f571931f429bac426bf194356600f9548332db91040f853895891bc41cd"} Nov 02 21:53:43.093587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:43.093447 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba70f571931f429bac426bf194356600f9548332db91040f853895891bc41cd" Nov 02 21:53:43.093587 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:43.093516 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb" Nov 02 21:53:45.942896 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:45.942863 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:45.966769 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:45.966748 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 21:53:49.800902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:49.800866 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw"] Nov 02 21:53:50.129830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:50.129789 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:50.257081 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:50.257026 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:50.619217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:50.619158 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:51.584322 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:51.584278 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:54.304314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:54.304280 4441 scope.go:122] "RemoveContainer" containerID="878af8762974757f1a7e930b2e75ab13d7080c56d31d8612c932f4dcec176193" Nov 02 21:53:54.341796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:54.341751 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:54.802039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:54.801992 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" podUID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" containerName="payload-processing" containerID="cri-o://ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea" gracePeriod=25 Nov 02 21:53:55.236756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:55.236709 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:55.433337 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:55.433290 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" podUID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:53:55.958946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:55.958914 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6"] Nov 02 21:53:55.959183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:55.959117 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" containerID="cri-o://38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8" gracePeriod=30 Nov 02 21:53:57.576473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:57.576437 4441 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:53:57.576871 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:57.576500 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:53:57.917405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:57.917317 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" containerID="cri-o://e26868832556b48b42ccdfeabe1de4b2d771260194f60d806eb56bbd791011b4" gracePeriod=45 Nov 02 21:53:57.937009 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:57.936949 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" containerID="cri-o://8c444be9ba9abc2c7f2244f7ba216f40aba1f710d1461141a2e98f1d6495c029" gracePeriod=45 Nov 02 21:53:58.028183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.028153 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/ready\": dial tcp 10.133.0.112:8000: connect: connection refused" Nov 02 21:53:58.042284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.042253 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Nov 02 21:53:58.225150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.225109 4441 generic.go:356] "Generic (PLEG): container finished" podID="8d6e43ff-e946-4b43-be35-672d616aac21" containerID="8c444be9ba9abc2c7f2244f7ba216f40aba1f710d1461141a2e98f1d6495c029" exitCode=0 Nov 02 21:53:58.225287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.225192 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" event={"ID":"8d6e43ff-e946-4b43-be35-672d616aac21","Type":"ContainerDied","Data":"8c444be9ba9abc2c7f2244f7ba216f40aba1f710d1461141a2e98f1d6495c029"} Nov 02 21:53:58.226936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.226913 4441 generic.go:356] "Generic (PLEG): container finished" podID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerID="e26868832556b48b42ccdfeabe1de4b2d771260194f60d806eb56bbd791011b4" exitCode=0 Nov 02 21:53:58.227085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.226952 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" event={"ID":"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c","Type":"ContainerDied","Data":"e26868832556b48b42ccdfeabe1de4b2d771260194f60d806eb56bbd791011b4"} Nov 02 21:53:58.227085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.227025 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" event={"ID":"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c","Type":"ContainerDied","Data":"a57f1abe5a5078f15aec1d1e0fa9eb4a1d0abc03370005fa692826e40e170946"} Nov 02 21:53:58.227085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.227043 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57f1abe5a5078f15aec1d1e0fa9eb4a1d0abc03370005fa692826e40e170946" Nov 02 21:53:58.239747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.239728 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:53:58.243350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.243331 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:53:58.313310 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313282 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-model-cache\") pod \"8d6e43ff-e946-4b43-be35-672d616aac21\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " Nov 02 21:53:58.313466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313340 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-tmp-dir\") pod \"8d6e43ff-e946-4b43-be35-672d616aac21\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " Nov 02 21:53:58.313466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313403 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-model-cache\") pod \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " Nov 02 21:53:58.313466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313425 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6e43ff-e946-4b43-be35-672d616aac21-kube-api-access-dpgcw\" (UniqueName: \"kubernetes.io/projected/8d6e43ff-e946-4b43-be35-672d616aac21-kube-api-access-dpgcw\") pod \"8d6e43ff-e946-4b43-be35-672d616aac21\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " Nov 02 21:53:58.313649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313476 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-home\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-home\") pod \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " Nov 02 21:53:58.313649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313581 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d6e43ff-e946-4b43-be35-672d616aac21-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6e43ff-e946-4b43-be35-672d616aac21-tls-certs\") pod \"8d6e43ff-e946-4b43-be35-672d616aac21\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " Nov 02 21:53:58.313649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313579 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-model-cache" pod "8d6e43ff-e946-4b43-be35-672d616aac21" (UID: "8d6e43ff-e946-4b43-be35-672d616aac21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.313649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313589 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-tmp-dir" pod "8d6e43ff-e946-4b43-be35-672d616aac21" (UID: "8d6e43ff-e946-4b43-be35-672d616aac21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.313649 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313629 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-dshm\") pod \"8d6e43ff-e946-4b43-be35-672d616aac21\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " Nov 02 21:53:58.313925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313793 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-home" pod "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" (UID: "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.313925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313833 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-dshm\") pod \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " Nov 02 21:53:58.313925 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313888 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-kube-api-access-dpmdj\" (UniqueName: \"kubernetes.io/projected/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-kube-api-access-dpmdj\") pod \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " Nov 02 21:53:58.314120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313934 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tls-certs\") pod \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " Nov 02 21:53:58.314120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313985 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-home\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-home\") pod \"8d6e43ff-e946-4b43-be35-672d616aac21\" (UID: \"8d6e43ff-e946-4b43-be35-672d616aac21\") " Nov 02 21:53:58.314120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.313992 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-model-cache" pod "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" (UID: "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.314120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314062 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tmp-dir\") pod \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\" (UID: \"3d2e45a9-0f78-485e-9f18-a3e376ca1e8c\") " Nov 02 21:53:58.314570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314537 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-home" pod "8d6e43ff-e946-4b43-be35-672d616aac21" (UID: "8d6e43ff-e946-4b43-be35-672d616aac21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.314774 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314750 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tmp-dir" pod "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" (UID: "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.314858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314761 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.314858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314806 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.314858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314822 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.314858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314836 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.314858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.314851 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.317403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.317351 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-dshm" pod "8d6e43ff-e946-4b43-be35-672d616aac21" (UID: "8d6e43ff-e946-4b43-be35-672d616aac21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.317505 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.317367 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6e43ff-e946-4b43-be35-672d616aac21-kube-api-access-dpgcw" pod "8d6e43ff-e946-4b43-be35-672d616aac21" (UID: "8d6e43ff-e946-4b43-be35-672d616aac21"). InnerVolumeSpecName "kube-api-access-dpgcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:53:58.323051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.321461 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d6e43ff-e946-4b43-be35-672d616aac21-tls-certs" pod "8d6e43ff-e946-4b43-be35-672d616aac21" (UID: "8d6e43ff-e946-4b43-be35-672d616aac21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:53:58.323051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.322709 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-kube-api-access-dpmdj" pod "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" (UID: "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c"). InnerVolumeSpecName "kube-api-access-dpmdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:53:58.323051 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.322728 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tls-certs" pod "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" (UID: "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:53:58.323216 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.323146 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-dshm" pod "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" (UID: "3d2e45a9-0f78-485e-9f18-a3e376ca1e8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.416519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.416485 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.416519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.416520 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmdj\" (UniqueName: \"kubernetes.io/projected/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-kube-api-access-dpmdj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.416732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.416534 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.416732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.416547 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.416732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.416559 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpgcw\" (UniqueName: \"kubernetes.io/projected/8d6e43ff-e946-4b43-be35-672d616aac21-kube-api-access-dpgcw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.416732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.416578 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6e43ff-e946-4b43-be35-672d616aac21-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.416732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:58.416589 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6e43ff-e946-4b43-be35-672d616aac21-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:59.238956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.238858 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" Nov 02 21:53:59.238956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.238875 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v" event={"ID":"8d6e43ff-e946-4b43-be35-672d616aac21","Type":"ContainerDied","Data":"705d978d4ae73c59594a336a812804e52363f02de51875ee19fd52832dbb20da"} Nov 02 21:53:59.238956 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.238932 4441 scope.go:122] "RemoveContainer" containerID="8c444be9ba9abc2c7f2244f7ba216f40aba1f710d1461141a2e98f1d6495c029" Nov 02 21:53:59.239543 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.239193 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6" Nov 02 21:53:59.395566 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.395533 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v"] Nov 02 21:53:59.399362 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.399339 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/simulator-w5-ef6b43-kserve-5698bf9d8b-kmz5v"] Nov 02 21:53:59.505859 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.505777 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6"] Nov 02 21:53:59.509678 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.509651 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-2-w5-ef6b43-kserve-789c66b849-tjfq6"] Nov 02 21:53:59.615127 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.615097 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:59.769018 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.768900 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:59.890390 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:53:59.890359 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:00.584414 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:00.584383 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" path="/var/lib/kubelet/pods/3d2e45a9-0f78-485e-9f18-a3e376ca1e8c/volumes" Nov 02 21:54:00.584840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:00.584826 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" path="/var/lib/kubelet/pods/8d6e43ff-e946-4b43-be35-672d616aac21/volumes" Nov 02 21:54:01.234193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.234122 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:54:01.257672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.257639 4441 generic.go:356] "Generic (PLEG): container finished" podID="d73714c5-4059-4c1f-924c-6883b6852311" containerID="38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8" exitCode=0 Nov 02 21:54:01.257799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.257691 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" Nov 02 21:54:01.257799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.257725 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" event={"ID":"d73714c5-4059-4c1f-924c-6883b6852311","Type":"ContainerDied","Data":"38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8"} Nov 02 21:54:01.257799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.257781 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6" event={"ID":"d73714c5-4059-4c1f-924c-6883b6852311","Type":"ContainerDied","Data":"8eb4a79f102ca1477c76e31c8eccc0bc4cc7b2388ed7d252865053afa80016e4"} Nov 02 21:54:01.257933 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.257815 4441 scope.go:122] "RemoveContainer" containerID="38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8" Nov 02 21:54:01.268086 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.268068 4441 scope.go:122] "RemoveContainer" containerID="38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8" Nov 02 21:54:01.268312 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:54:01.268294 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8\": container with ID starting with 38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8 not found: ID does not exist" containerID="38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8" Nov 02 21:54:01.268362 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.268320 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8"} err="failed to get container status \"38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8\": rpc error: code = NotFound desc = could not find container \"38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8\": container with ID starting with 38b2539eb0bdfb08a1e106899e880be6b3a4f584c5f2d8a79709dc7b14d6cda8 not found: ID does not exist" Nov 02 21:54:01.358716 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.358682 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d73714c5-4059-4c1f-924c-6883b6852311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73714c5-4059-4c1f-924c-6883b6852311-istio-podinfo\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.358910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.358776 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-credential-socket\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.358910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.358837 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-socket\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.358910 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.358884 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-envoy\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.359151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.358914 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-istio-token\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-istio-token\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.359151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359004 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-certs\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.359151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359045 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73714c5-4059-4c1f-924c-6883b6852311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73714c5-4059-4c1f-924c-6883b6852311-istiod-ca-cert\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.359151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359103 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-kube-api-access-6d5kv\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.359151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359147 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-data\") pod \"d73714c5-4059-4c1f-924c-6883b6852311\" (UID: \"d73714c5-4059-4c1f-924c-6883b6852311\") " Nov 02 21:54:01.359424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359203 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-socket" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:01.359424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359221 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-credential-socket" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:01.359541 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359434 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-certs" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:01.359541 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359506 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73714c5-4059-4c1f-924c-6883b6852311-istiod-ca-cert" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:01.359632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359612 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-data" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:01.359863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359842 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.360001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359866 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73714c5-4059-4c1f-924c-6883b6852311-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.360001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359881 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.360001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359896 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.360001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.359910 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.361422 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.361395 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-istio-token" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:01.361567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.361546 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-envoy" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:01.361791 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.361764 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d73714c5-4059-4c1f-924c-6883b6852311-istio-podinfo" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:54:01.361874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.361857 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-kube-api-access-6d5kv" pod "d73714c5-4059-4c1f-924c-6883b6852311" (UID: "d73714c5-4059-4c1f-924c-6883b6852311"). InnerVolumeSpecName "kube-api-access-6d5kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:01.461319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.461279 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73714c5-4059-4c1f-924c-6883b6852311-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.461319 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.461319 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73714c5-4059-4c1f-924c-6883b6852311-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.461597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.461333 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.461597 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.461346 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/d73714c5-4059-4c1f-924c-6883b6852311-kube-api-access-6d5kv\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:01.677049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.677018 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6"] Nov 02 21:54:01.681108 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:01.681077 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-7ntr6"] Nov 02 21:54:02.586937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:02.586900 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73714c5-4059-4c1f-924c-6883b6852311" path="/var/lib/kubelet/pods/d73714c5-4059-4c1f-924c-6883b6852311/volumes" Nov 02 21:54:04.401451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:04.401402 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:05.433492 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:05.433452 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" podUID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:54:08.990355 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:08.990317 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:10.615502 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:54:10.615467 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:54:13.486866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:13.486834 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:15.087439 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.087413 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:54:15.232031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.231999 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-kube-api-access-8lxjw\" (UniqueName: \"kubernetes.io/projected/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-kube-api-access-8lxjw\") pod \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " Nov 02 21:54:15.232233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.232155 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-plugins-config-volume\") pod \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\" (UID: \"a0c402d8-c4e6-4fb2-b43c-c360fa3da221\") " Nov 02 21:54:15.232600 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.232570 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-plugins-config-volume" pod "a0c402d8-c4e6-4fb2-b43c-c360fa3da221" (UID: "a0c402d8-c4e6-4fb2-b43c-c360fa3da221"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:15.232750 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.232731 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:15.234815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.234792 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-kube-api-access-8lxjw" pod "a0c402d8-c4e6-4fb2-b43c-c360fa3da221" (UID: "a0c402d8-c4e6-4fb2-b43c-c360fa3da221"). InnerVolumeSpecName "kube-api-access-8lxjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:15.333758 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.333676 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lxjw\" (UniqueName: \"kubernetes.io/projected/a0c402d8-c4e6-4fb2-b43c-c360fa3da221-kube-api-access-8lxjw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:15.373202 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.373165 4441 generic.go:356] "Generic (PLEG): container finished" podID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" containerID="ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea" exitCode=0 Nov 02 21:54:15.373364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.373213 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" Nov 02 21:54:15.373364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.373250 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" event={"ID":"a0c402d8-c4e6-4fb2-b43c-c360fa3da221","Type":"ContainerDied","Data":"ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea"} Nov 02 21:54:15.373364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.373310 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw" event={"ID":"a0c402d8-c4e6-4fb2-b43c-c360fa3da221","Type":"ContainerDied","Data":"73f28beb482f1320b06684315719b4237173f0a69089d2ab728fe1e5a35b3b44"} Nov 02 21:54:15.373364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.373341 4441 scope.go:122] "RemoveContainer" containerID="ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea" Nov 02 21:54:15.385037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.385016 4441 scope.go:122] "RemoveContainer" containerID="ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea" Nov 02 21:54:15.385351 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:54:15.385328 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea\": container with ID starting with ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea not found: ID does not exist" containerID="ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea" Nov 02 21:54:15.385453 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.385358 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea"} err="failed to get container status \"ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea\": rpc error: code = NotFound desc = could not find container \"ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea\": container with ID starting with ec6a19fc8c7c522afc8c32ff2d7578d7dc727f9507aaf3c8dae40c551d7e11ea not found: ID does not exist" Nov 02 21:54:15.531145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.531111 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw"] Nov 02 21:54:15.534797 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:15.534773 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6b43-74fb5cfdbf-n4kdw"] Nov 02 21:54:16.588418 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:16.588378 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" path="/var/lib/kubelet/pods/a0c402d8-c4e6-4fb2-b43c-c360fa3da221/volumes" Nov 02 21:54:17.955614 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:17.955577 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:19.114475 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:19.114443 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:19.242927 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:19.242896 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:21.084134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:21.084098 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:21.601852 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:21.601817 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:23.277991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:23.277506 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:23.670993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:23.662074 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:23.708296 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:23.708252 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:23.899992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:23.899080 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:23.917990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:23.917047 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:23.952677 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:23.952581 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:23.975490 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:23.975443 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.020124 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.020075 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.045125 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.045079 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.055111 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.055052 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.902140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.902095 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.903576 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.903543 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:54:24.920166 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.920114 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.921163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.921145 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:54:24.953084 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.953041 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.953610 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.953588 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:54:24.977075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.977037 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" podUID="1ca7a877-0358-45c1-949e-b32728684968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:24.977638 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:24.977614 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:54:25.025671 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.025634 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:54:25.026323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.026297 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:54:25.052233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.052190 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:25.090687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.081195 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:25.097996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.091197 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:54:25.098351 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.098334 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:54:25.130288 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.104653 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:25.320044 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.304091 4441 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:25.624849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.623733 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh" Nov 02 21:54:25.625462 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.625348 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 21:54:25.642027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.633210 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 21:54:25.642027 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.635848 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 21:54:25.729221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.729154 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 21:54:25.733388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.732790 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 21:54:25.871931 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:25.863128 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 21:54:27.200783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:27.200748 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:27.330993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:27.329948 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:27.910990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:27.910639 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:32.385595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:32.385560 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:32.600124 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:32.600091 4441 patch_prober.go:29] interesting pod/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:54:32.600324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:32.600152 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:34.053359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.053329 4441 generic.go:356] "Generic (PLEG): container finished" podID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerID="5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655" exitCode=0 Nov 02 21:54:34.054011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.053409 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" event={"ID":"40c41f4c-73ee-440d-befc-9def36a8b0ad","Type":"ContainerDied","Data":"5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655"} Nov 02 21:54:34.054011 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.053890 4441 scope.go:122] "RemoveContainer" containerID="5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655" Nov 02 21:54:34.055329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.055306 4441 generic.go:356] "Generic (PLEG): container finished" podID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerID="cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2" exitCode=0 Nov 02 21:54:34.055440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.055348 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" event={"ID":"3e865ee9-3056-48a2-bed0-19b609cb1795","Type":"ContainerDied","Data":"cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2"} Nov 02 21:54:34.055602 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.055576 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:54:34.055672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.055635 4441 scope.go:122] "RemoveContainer" containerID="cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2" Nov 02 21:54:34.055672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.055663 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:54:34.055792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.055713 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:54:34.057047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.057023 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:54:34.057114 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.057084 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:54:34.057156 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:34.057114 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:54:35.064701 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:35.064665 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" event={"ID":"40c41f4c-73ee-440d-befc-9def36a8b0ad","Type":"ContainerStarted","Data":"edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218"} Nov 02 21:54:35.066907 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:35.066877 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" event={"ID":"3e865ee9-3056-48a2-bed0-19b609cb1795","Type":"ContainerStarted","Data":"3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7"} Nov 02 21:54:35.421122 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:35.421030 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:54:35.421400 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:35.421366 4441 patch_prober.go:29] interesting pod/e2e-inf-b-70d237-openshift-default-86895f855-4l98n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 21:54:35.421481 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:35.421453 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 21:54:36.059295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.059261 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:54:36.059667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.059641 4441 patch_prober.go:29] interesting pod/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Nov 02 21:54:36.059793 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.059709 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Nov 02 21:54:36.348048 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.347457 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk"] Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367864 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367894 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367911 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367918 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367942 4441 cpu_manager.go:414] "containerMap: removing container" podUID="5d394cb1-c26f-4351-8d50-fa991b38e8f1" containerName="revoke-keys" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367951 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d394cb1-c26f-4351-8d50-fa991b38e8f1" containerName="revoke-keys" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367960 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" containerName="payload-processing" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.367983 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" containerName="payload-processing" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.368023 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.368032 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.368207 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2e45a9-0f78-485e-9f18-a3e376ca1e8c" containerName="main" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.368219 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73714c5-4059-4c1f-924c-6883b6852311" containerName="istio-proxy" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.368231 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d394cb1-c26f-4351-8d50-fa991b38e8f1" containerName="revoke-keys" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.368240 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c402d8-c4e6-4fb2-b43c-c360fa3da221" containerName="payload-processing" Nov 02 21:54:36.369002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.368256 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6e43ff-e946-4b43-be35-672d616aac21" containerName="main" Nov 02 21:54:36.373001 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.371890 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.428991 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.423543 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:54:36.442378 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.436121 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:36.443829 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.436186 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:36.457119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.442336 4441 patch_prober.go:29] interesting pod/e2e-inf-b-70d237-openshift-default-86895f855-4l98n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 21:54:36.457313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.457289 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 21:54:36.462028 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.461727 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs"] Nov 02 21:54:36.462698 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.462600 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:36.465989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.465955 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.472987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.468667 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk"] Nov 02 21:54:36.472987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.468668 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmg4\" (UniqueName: \"kubernetes.io/projected/841ecdc8-337b-4e93-84cf-606cfb161a88-kube-api-access-vdmg4\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.472987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.468757 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.472987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.468822 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.472987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.468841 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jczv\" (UniqueName: \"kubernetes.io/projected/e6c27d6a-53df-4736-a77e-b59b60c828e6-kube-api-access-9jczv\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.472987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.468927 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.472987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.468942 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.544241 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.544158 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs"] Nov 02 21:54:36.622050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.598744 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmg4\" (UniqueName: \"kubernetes.io/projected/841ecdc8-337b-4e93-84cf-606cfb161a88-kube-api-access-vdmg4\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.622050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.598827 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.622050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.598932 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.622050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.598984 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jczv\" (UniqueName: \"kubernetes.io/projected/e6c27d6a-53df-4736-a77e-b59b60c828e6-kube-api-access-9jczv\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.622050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.599134 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.622050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.599164 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.633237 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.633200 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.638196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.635384 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.644987 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.643006 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.645230 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.645207 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.696990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.696934 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmg4\" (UniqueName: \"kubernetes.io/projected/841ecdc8-337b-4e93-84cf-606cfb161a88-kube-api-access-vdmg4\") pod \"maas-api-revoke-sub-18988557-cznvs\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:36.707989 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.704898 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jczv\" (UniqueName: \"kubernetes.io/projected/e6c27d6a-53df-4736-a77e-b59b60c828e6-kube-api-access-9jczv\") pod \"maas-api-revoke-sub-310b7510-gq7vk\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.758915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.758843 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:36.847539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:36.847511 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:37.059232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:37.059205 4441 patch_prober.go:29] interesting pod/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Nov 02 21:54:37.059376 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:37.059253 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Nov 02 21:54:37.229045 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:37.226824 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:37.417101 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:37.417045 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk"] Nov 02 21:54:37.429153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:37.423151 4441 patch_prober.go:29] interesting pod/e2e-inf-b-70d237-openshift-default-86895f855-4l98n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 21:54:37.429153 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:37.423209 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 21:54:37.440936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:37.440901 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs"] Nov 02 21:54:38.060146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.060050 4441 patch_prober.go:29] interesting pod/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Nov 02 21:54:38.060146 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.060103 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Nov 02 21:54:38.154042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.153997 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" event={"ID":"e6c27d6a-53df-4736-a77e-b59b60c828e6","Type":"ContainerStarted","Data":"0f324887b5fbb655fb94d8cc433224f81e08728ae7b6498a986ba85eb4d77d29"} Nov 02 21:54:38.154247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.154058 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" event={"ID":"e6c27d6a-53df-4736-a77e-b59b60c828e6","Type":"ContainerStarted","Data":"5f8b86b19ce3e9de6960ec912e29529328278da31c9a3b181a62bc6f05b3025a"} Nov 02 21:54:38.160334 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.160218 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" event={"ID":"841ecdc8-337b-4e93-84cf-606cfb161a88","Type":"ContainerStarted","Data":"d2ce5f4ad9672787ba4c56b4884baf42f50162903a5cdde607f41415aac5cfb6"} Nov 02 21:54:38.160480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.160350 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" event={"ID":"841ecdc8-337b-4e93-84cf-606cfb161a88","Type":"ContainerStarted","Data":"b529e5f5c36f34881d5422e1b95f1545c6e1aef3f23783bcb755476bdecdfa04"} Nov 02 21:54:38.308990 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.302745 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" podStartSLOduration=2.302726255 podStartE2EDuration="2.302726255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:38.291116076 +0000 UTC m=+2507.846744414" watchObservedRunningTime="2026-11-02 21:54:38.302726255 +0000 UTC m=+2507.858354594" Nov 02 21:54:38.427067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.426462 4441 patch_prober.go:29] interesting pod/e2e-inf-b-70d237-openshift-default-86895f855-4l98n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 21:54:38.427067 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.426523 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 21:54:38.429060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:38.428957 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" podStartSLOduration=2.428938834 podStartE2EDuration="2.428938834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:38.403448623 +0000 UTC m=+2507.959076964" watchObservedRunningTime="2026-11-02 21:54:38.428938834 +0000 UTC m=+2507.984567172" Nov 02 21:54:39.059408 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.059374 4441 patch_prober.go:29] interesting pod/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Nov 02 21:54:39.059567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.059430 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Nov 02 21:54:39.104992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.104055 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:54:39.172993 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.172933 4441 generic.go:356] "Generic (PLEG): container finished" podID="e6c27d6a-53df-4736-a77e-b59b60c828e6" containerID="0f324887b5fbb655fb94d8cc433224f81e08728ae7b6498a986ba85eb4d77d29" exitCode=0 Nov 02 21:54:39.173717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.173319 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" event={"ID":"e6c27d6a-53df-4736-a77e-b59b60c828e6","Type":"ContainerDied","Data":"0f324887b5fbb655fb94d8cc433224f81e08728ae7b6498a986ba85eb4d77d29"} Nov 02 21:54:39.184821 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.184587 4441 generic.go:356] "Generic (PLEG): container finished" podID="841ecdc8-337b-4e93-84cf-606cfb161a88" containerID="d2ce5f4ad9672787ba4c56b4884baf42f50162903a5cdde607f41415aac5cfb6" exitCode=0 Nov 02 21:54:39.184821 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.184670 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" event={"ID":"841ecdc8-337b-4e93-84cf-606cfb161a88","Type":"ContainerDied","Data":"d2ce5f4ad9672787ba4c56b4884baf42f50162903a5cdde607f41415aac5cfb6"} Nov 02 21:54:39.420423 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.420336 4441 patch_prober.go:29] interesting pod/e2e-inf-b-70d237-openshift-default-86895f855-4l98n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 21:54:39.420700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:39.420672 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 21:54:40.058692 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.058659 4441 patch_prober.go:29] interesting pod/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Nov 02 21:54:40.059130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.058943 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Nov 02 21:54:40.427572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.427086 4441 patch_prober.go:29] interesting pod/e2e-inf-b-70d237-openshift-default-86895f855-4l98n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 21:54:40.427572 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.427152 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 21:54:40.551711 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.551684 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:40.555405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.555382 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:40.625199 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:54:40.625100 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:54:40.642178 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.642137 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-xks-ca\") pod \"841ecdc8-337b-4e93-84cf-606cfb161a88\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " Nov 02 21:54:40.642394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.642236 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-xks-ca\") pod \"e6c27d6a-53df-4736-a77e-b59b60c828e6\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " Nov 02 21:54:40.642394 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.642361 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-openshift-ca\") pod \"841ecdc8-337b-4e93-84cf-606cfb161a88\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " Nov 02 21:54:40.642533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.642397 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841ecdc8-337b-4e93-84cf-606cfb161a88-kube-api-access-vdmg4\" (UniqueName: \"kubernetes.io/projected/841ecdc8-337b-4e93-84cf-606cfb161a88-kube-api-access-vdmg4\") pod \"841ecdc8-337b-4e93-84cf-606cfb161a88\" (UID: \"841ecdc8-337b-4e93-84cf-606cfb161a88\") " Nov 02 21:54:40.642533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.642451 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-openshift-ca\") pod \"e6c27d6a-53df-4736-a77e-b59b60c828e6\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " Nov 02 21:54:40.642533 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.642500 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c27d6a-53df-4736-a77e-b59b60c828e6-kube-api-access-9jczv\" (UniqueName: \"kubernetes.io/projected/e6c27d6a-53df-4736-a77e-b59b60c828e6-kube-api-access-9jczv\") pod \"e6c27d6a-53df-4736-a77e-b59b60c828e6\" (UID: \"e6c27d6a-53df-4736-a77e-b59b60c828e6\") " Nov 02 21:54:40.648519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.648193 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-openshift-ca" pod "841ecdc8-337b-4e93-84cf-606cfb161a88" (UID: "841ecdc8-337b-4e93-84cf-606cfb161a88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:40.650138 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.650106 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-xks-ca" pod "841ecdc8-337b-4e93-84cf-606cfb161a88" (UID: "841ecdc8-337b-4e93-84cf-606cfb161a88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:40.650622 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.650567 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-openshift-ca" pod "e6c27d6a-53df-4736-a77e-b59b60c828e6" (UID: "e6c27d6a-53df-4736-a77e-b59b60c828e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:40.653947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.653917 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c27d6a-53df-4736-a77e-b59b60c828e6-kube-api-access-9jczv" pod "e6c27d6a-53df-4736-a77e-b59b60c828e6" (UID: "e6c27d6a-53df-4736-a77e-b59b60c828e6"). InnerVolumeSpecName "kube-api-access-9jczv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:40.667211 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.667153 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-xks-ca" pod "e6c27d6a-53df-4736-a77e-b59b60c828e6" (UID: "e6c27d6a-53df-4736-a77e-b59b60c828e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:40.669599 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.669567 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841ecdc8-337b-4e93-84cf-606cfb161a88-kube-api-access-vdmg4" pod "841ecdc8-337b-4e93-84cf-606cfb161a88" (UID: "841ecdc8-337b-4e93-84cf-606cfb161a88"). InnerVolumeSpecName "kube-api-access-vdmg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:40.708766 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.707818 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:40.744306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.744274 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:40.744306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.744300 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdmg4\" (UniqueName: \"kubernetes.io/projected/841ecdc8-337b-4e93-84cf-606cfb161a88-kube-api-access-vdmg4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:40.744306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.744311 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:40.744591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.744320 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jczv\" (UniqueName: \"kubernetes.io/projected/e6c27d6a-53df-4736-a77e-b59b60c828e6-kube-api-access-9jczv\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:40.744591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.744329 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841ecdc8-337b-4e93-84cf-606cfb161a88-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:40.744591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:40.744337 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c27d6a-53df-4736-a77e-b59b60c828e6-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:41.063915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.063888 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:54:41.064413 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.064395 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:54:41.065102 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.065079 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 21:54:41.234496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.234463 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" Nov 02 21:54:41.234667 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.234460 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk" event={"ID":"e6c27d6a-53df-4736-a77e-b59b60c828e6","Type":"ContainerDied","Data":"5f8b86b19ce3e9de6960ec912e29529328278da31c9a3b181a62bc6f05b3025a"} Nov 02 21:54:41.234764 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.234663 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f8b86b19ce3e9de6960ec912e29529328278da31c9a3b181a62bc6f05b3025a" Nov 02 21:54:41.236022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.235996 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" event={"ID":"841ecdc8-337b-4e93-84cf-606cfb161a88","Type":"ContainerDied","Data":"b529e5f5c36f34881d5422e1b95f1545c6e1aef3f23783bcb755476bdecdfa04"} Nov 02 21:54:41.236158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.236027 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs" Nov 02 21:54:41.236158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.236040 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b529e5f5c36f34881d5422e1b95f1545c6e1aef3f23783bcb755476bdecdfa04" Nov 02 21:54:41.424473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.424396 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:54:41.425584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.425567 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 21:54:41.978427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:41.978389 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:42.527761 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.527728 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c"] Nov 02 21:54:42.528343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.528329 4441 cpu_manager.go:414] "containerMap: removing container" podUID="841ecdc8-337b-4e93-84cf-606cfb161a88" containerName="revoke-keys" Nov 02 21:54:42.528447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.528343 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841ecdc8-337b-4e93-84cf-606cfb161a88" containerName="revoke-keys" Nov 02 21:54:42.528447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.528359 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e6c27d6a-53df-4736-a77e-b59b60c828e6" containerName="revoke-keys" Nov 02 21:54:42.528447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.528364 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c27d6a-53df-4736-a77e-b59b60c828e6" containerName="revoke-keys" Nov 02 21:54:42.528561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.528521 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="841ecdc8-337b-4e93-84cf-606cfb161a88" containerName="revoke-keys" Nov 02 21:54:42.528561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.528533 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c27d6a-53df-4736-a77e-b59b60c828e6" containerName="revoke-keys" Nov 02 21:54:42.531121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.531104 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.583299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.582461 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:42.583299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.582649 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:42.583299 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.582705 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:42.593278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.593250 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c"] Nov 02 21:54:42.626343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.626315 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:54:42.670757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.670715 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.671119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.671093 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.671224 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.671135 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqnh\" (UniqueName: \"kubernetes.io/projected/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-kube-api-access-msqnh\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.772814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.772778 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.772814 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.772817 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqnh\" (UniqueName: \"kubernetes.io/projected/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-kube-api-access-msqnh\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.773121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.772919 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.773733 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.773698 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.775306 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.775278 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.827751 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.827721 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqnh\" (UniqueName: \"kubernetes.io/projected/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-kube-api-access-msqnh\") pod \"maas-api-revoke-keys-e2e-api-a-d0946d-nf62c\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:42.846586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:42.846564 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:43.233519 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:43.233487 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c"] Nov 02 21:54:43.234360 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:54:43.234324 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca363a0_c3bc_4ca8_95b1_d9ecf3a6164e.slice/crio-5c8a021e081995d2a935d134e34242955476462711efe5b66bef14121231fe62 WatchSource:0}: Error finding container 5c8a021e081995d2a935d134e34242955476462711efe5b66bef14121231fe62: Status 404 returned error can't find the container with id 5c8a021e081995d2a935d134e34242955476462711efe5b66bef14121231fe62 Nov 02 21:54:43.257669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:43.257641 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" event={"ID":"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e","Type":"ContainerStarted","Data":"5c8a021e081995d2a935d134e34242955476462711efe5b66bef14121231fe62"} Nov 02 21:54:44.267219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:44.267177 4441 generic.go:356] "Generic (PLEG): container finished" podID="4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" containerID="91cc12317461a80d4546d33f47500e72410a71fee95db7a82b75094847fd6f64" exitCode=0 Nov 02 21:54:44.267618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:44.267219 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" event={"ID":"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e","Type":"ContainerDied","Data":"91cc12317461a80d4546d33f47500e72410a71fee95db7a82b75094847fd6f64"} Nov 02 21:54:45.434496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.434472 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:45.610547 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.610501 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-kube-api-access-msqnh\" (UniqueName: \"kubernetes.io/projected/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-kube-api-access-msqnh\") pod \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " Nov 02 21:54:45.610726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.610569 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-xks-ca\") pod \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " Nov 02 21:54:45.610781 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.610744 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-openshift-ca\") pod \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\" (UID: \"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e\") " Nov 02 21:54:45.611115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.611088 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-openshift-ca" pod "4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" (UID: "4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:45.611371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.611351 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:45.612743 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.612718 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-kube-api-access-msqnh" pod "4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" (UID: "4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e"). InnerVolumeSpecName "kube-api-access-msqnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:45.612743 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.612734 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-xks-ca" pod "4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" (UID: "4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:45.712500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.712469 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqnh\" (UniqueName: \"kubernetes.io/projected/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-kube-api-access-msqnh\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:45.712500 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:45.712497 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:46.285757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:46.285704 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" Nov 02 21:54:46.285757 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:46.285720 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c" event={"ID":"4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e","Type":"ContainerDied","Data":"5c8a021e081995d2a935d134e34242955476462711efe5b66bef14121231fe62"} Nov 02 21:54:46.286037 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:46.285773 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c8a021e081995d2a935d134e34242955476462711efe5b66bef14121231fe62" Nov 02 21:54:46.738002 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:46.737947 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:51.388022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:51.387962 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:51.548443 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:51.548402 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:53.463294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:53.463263 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:53.587252 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:53.587214 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:54.337017 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:54.336981 4441 scope.go:122] "RemoveContainer" containerID="fc573bcef3d88a428a21b7e86688ff4060fcb492f97f58aa0749127d34d36710" Nov 02 21:54:54.348195 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:54.348172 4441 scope.go:122] "RemoveContainer" containerID="8bd3dfc2193e8021007db8f9e6e8326a657e17493257731a217852691749b82c" Nov 02 21:54:54.359427 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:54.359403 4441 scope.go:122] "RemoveContainer" containerID="e26868832556b48b42ccdfeabe1de4b2d771260194f60d806eb56bbd791011b4" Nov 02 21:54:54.370560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:54.370538 4441 scope.go:122] "RemoveContainer" containerID="446c068ac34ab436c3dab8637e45fa313510a250e8982fc5137361524f7a459e" Nov 02 21:54:54.380070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:54.380050 4441 scope.go:122] "RemoveContainer" containerID="940baa1c365570205a8774b474278f9d221a52ea0a38846ceaddf05b6292ba13" Nov 02 21:54:54.708031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:54.707910 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:56.180385 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:54:56.180350 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:00.739695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:00.739657 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:05.382858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:05.382820 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:07.084748 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:07.084712 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:10.316441 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:10.316408 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:10.615653 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:55:10.615572 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:55:15.091183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:15.091129 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:19.063625 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:19.063590 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:19.199936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:19.199906 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:19.582641 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:19.582608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:24.045115 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:24.045079 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:28.617480 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:28.617444 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:32.385865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:32.385783 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:33.050024 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:33.049987 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552"] Nov 02 21:55:33.097034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:33.097005 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk"] Nov 02 21:55:33.100893 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:33.100867 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2fc43d-65552"] Nov 02 21:55:33.104266 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:33.104241 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146703ee-scclk"] Nov 02 21:55:33.496150 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:33.496117 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:34.231384 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:34.231352 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:34.584887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:34.584857 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3789c0de-a398-48b6-ac6e-09b053059650" path="/var/lib/kubelet/pods/3789c0de-a398-48b6-ac6e-09b053059650/volumes" Nov 02 21:55:34.585402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:34.585386 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48e7c50-073c-4d01-a89e-31b107577c1e" path="/var/lib/kubelet/pods/c48e7c50-073c-4d01-a89e-31b107577c1e/volumes" Nov 02 21:55:37.991092 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:37.991050 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:40.616449 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:55:40.616411 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:55:42.596881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:42.596827 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:47.179448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:47.179409 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:51.463286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:51.463246 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:51.576869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:51.576821 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:51.712704 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:51.712648 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:55.037424 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:55.037387 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:55.579874 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:55.579835 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:57.054137 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:57.054101 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:57.275300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:57.275251 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:57.397919 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:57.397864 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:57.667193 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:57.667114 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb"] Nov 02 21:55:57.713743 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:57.713713 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6b43-hqwnb"] Nov 02 21:55:58.095812 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:58.095777 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:58.584402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:55:58.584374 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d394cb1-c26f-4351-8d50-fa991b38e8f1" path="/var/lib/kubelet/pods/5d394cb1-c26f-4351-8d50-fa991b38e8f1/volumes" Nov 02 21:56:02.059286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:02.059255 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:04.024460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:04.024417 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:04.147501 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:04.147471 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:04.338895 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:04.338862 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj"] Nov 02 21:56:04.339152 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:04.339127 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" containerID="cri-o://a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1" gracePeriod=30 Nov 02 21:56:04.345383 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:04.345356 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:06.603567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:06.603531 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:09.623265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.623201 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:56:09.782851 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.782808 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-data\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.783460 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783352 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-data" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783454 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-podinfo\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783536 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-envoy\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783570 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-credential-socket\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783598 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-kube-api-access-8mggk\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-kube-api-access-8mggk\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783642 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff4fe47-15c6-456b-a5d4-431f95324537-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff4fe47-15c6-456b-a5d4-431f95324537-istiod-ca-cert\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783786 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-token\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783819 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-socket\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784050 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.783887 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-certs\") pod \"6ff4fe47-15c6-456b-a5d4-431f95324537\" (UID: \"6ff4fe47-15c6-456b-a5d4-431f95324537\") " Nov 02 21:56:09.784604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.784273 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff4fe47-15c6-456b-a5d4-431f95324537-istiod-ca-cert" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:09.784604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.784494 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff4fe47-15c6-456b-a5d4-431f95324537-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.784604 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.784514 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.784830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.784807 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-credential-socket" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:09.785560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.785523 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-socket" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:09.785720 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.785695 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-certs" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:09.787353 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.787327 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-podinfo" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:56:09.788089 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.788065 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-envoy" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:09.788647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.788627 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-token" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:09.790627 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.790606 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-kube-api-access-8mggk" pod "6ff4fe47-15c6-456b-a5d4-431f95324537" (UID: "6ff4fe47-15c6-456b-a5d4-431f95324537"). InnerVolumeSpecName "kube-api-access-8mggk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:09.886219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.886153 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.886219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.886186 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.886219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.886200 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.886219 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.886213 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.886447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.886226 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.886447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.886240 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff4fe47-15c6-456b-a5d4-431f95324537-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:09.886447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:09.886254 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mggk\" (UniqueName: \"kubernetes.io/projected/6ff4fe47-15c6-456b-a5d4-431f95324537-kube-api-access-8mggk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:56:10.137630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.137525 4441 generic.go:356] "Generic (PLEG): container finished" podID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerID="a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1" exitCode=0 Nov 02 21:56:10.137630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.137588 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" Nov 02 21:56:10.137630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.137608 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" event={"ID":"6ff4fe47-15c6-456b-a5d4-431f95324537","Type":"ContainerDied","Data":"a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1"} Nov 02 21:56:10.137957 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.137699 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj" event={"ID":"6ff4fe47-15c6-456b-a5d4-431f95324537","Type":"ContainerDied","Data":"070429a078300f632c50c332ad25a50a4830ee134d9fc4cdec93531c288ac645"} Nov 02 21:56:10.137957 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.137716 4441 scope.go:122] "RemoveContainer" containerID="a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1" Nov 02 21:56:10.148838 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.148816 4441 scope.go:122] "RemoveContainer" containerID="291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788" Nov 02 21:56:10.158687 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.158665 4441 scope.go:122] "RemoveContainer" containerID="a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1" Nov 02 21:56:10.158983 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:56:10.158937 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1\": container with ID starting with a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1 not found: ID does not exist" containerID="a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1" Nov 02 21:56:10.159060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.158989 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1"} err="failed to get container status \"a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1\": rpc error: code = NotFound desc = could not find container \"a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1\": container with ID starting with a0eddbb3d21cde19aa16096c59b823eb70516c174a27ce101d85c9e49afb1cb1 not found: ID does not exist" Nov 02 21:56:10.159060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.159008 4441 scope.go:122] "RemoveContainer" containerID="291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788" Nov 02 21:56:10.159323 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:56:10.159300 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788\": container with ID starting with 291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788 not found: ID does not exist" containerID="291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788" Nov 02 21:56:10.159454 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.159324 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788"} err="failed to get container status \"291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788\": rpc error: code = NotFound desc = could not find container \"291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788\": container with ID starting with 291b2bf1d0f8eb6e75bf53faf1d532393dff5ae33d75e381dcad1800fb8b4788 not found: ID does not exist" Nov 02 21:56:10.173651 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:56:10.173629 4441 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff4fe47_15c6_456b_a5d4_431f95324537.slice/crio-070429a078300f632c50c332ad25a50a4830ee134d9fc4cdec93531c288ac645\": RecentStats: unable to find data in memory cache]" Nov 02 21:56:10.296242 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.296203 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj"] Nov 02 21:56:10.300031 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.300002 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-cl9rj"] Nov 02 21:56:10.584220 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:10.584187 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" path="/var/lib/kubelet/pods/6ff4fe47-15c6-456b-a5d4-431f95324537/volumes" Nov 02 21:56:10.618369 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:56:10.618328 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:56:11.091580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:11.091546 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:15.409517 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.409482 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4"] Nov 02 21:56:15.410192 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410174 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" Nov 02 21:56:15.410253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410193 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" Nov 02 21:56:15.410253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410207 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" Nov 02 21:56:15.410253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410215 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" Nov 02 21:56:15.410253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410242 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" containerName="revoke-keys" Nov 02 21:56:15.410253 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410247 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" containerName="revoke-keys" Nov 02 21:56:15.410431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410369 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" Nov 02 21:56:15.410431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410378 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff4fe47-15c6-456b-a5d4-431f95324537" containerName="istio-proxy" Nov 02 21:56:15.410431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.410387 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" containerName="revoke-keys" Nov 02 21:56:15.415484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.415463 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.466930 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.466896 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4"] Nov 02 21:56:15.505245 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.505194 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:15.508034 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.508008 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-11537e\"/\"openshift-service-ca.crt\"" Nov 02 21:56:15.508170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.508034 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-11537e\"/\"kube-root-ca.crt\"" Nov 02 21:56:15.512340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.512316 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-11537e\"/\"default-dockercfg-p7jhj\"" Nov 02 21:56:15.512458 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.512411 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-11537e\"/\"test-model-11537e-kserve-self-signed-certs\"" Nov 02 21:56:15.575994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.563747 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-model-cache\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.575994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.563865 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-home\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.575994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.564014 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-dshm\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.575994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.564049 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhjn\" (UniqueName: \"kubernetes.io/projected/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-kube-api-access-dfhjn\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.575994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.564093 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tls-certs\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.575994 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.564167 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tmp-dir\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665426 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665342 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-home\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665441 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-dshm\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665467 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhjn\" (UniqueName: \"kubernetes.io/projected/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-kube-api-access-dfhjn\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665495 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tls-certs\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665542 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tmp-dir\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665574 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-model-cache\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665799 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-home\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.665923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665894 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tmp-dir\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.666324 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.665924 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-model-cache\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.667744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.667723 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-dshm\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.667996 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.667952 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tls-certs\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.689248 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.689216 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:15.719744 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.719708 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhjn\" (UniqueName: \"kubernetes.io/projected/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-kube-api-access-dfhjn\") pod \"test-model-11537e-kserve-566cb5694c-btql4\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:15.728495 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:15.728469 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:16.109523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:16.109490 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4"] Nov 02 21:56:16.110826 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:56:16.110795 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7e53ada_cf10_4902_9c6d_6bb378fc87ce.slice/crio-2cd40af9cacc75bcd4d33ea8b26c925f82c4313d250e1b869c4294bc36201d6f WatchSource:0}: Error finding container 2cd40af9cacc75bcd4d33ea8b26c925f82c4313d250e1b869c4294bc36201d6f: Status 404 returned error can't find the container with id 2cd40af9cacc75bcd4d33ea8b26c925f82c4313d250e1b869c4294bc36201d6f Nov 02 21:56:16.199033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:16.198998 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" event={"ID":"e7e53ada-cf10-4902-9c6d-6bb378fc87ce","Type":"ContainerStarted","Data":"e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6"} Nov 02 21:56:16.199222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:16.199045 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" event={"ID":"e7e53ada-cf10-4902-9c6d-6bb378fc87ce","Type":"ContainerStarted","Data":"2cd40af9cacc75bcd4d33ea8b26c925f82c4313d250e1b869c4294bc36201d6f"} Nov 02 21:56:16.309184 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:16.309131 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" podStartSLOduration=1.30911177 podStartE2EDuration="1.30911177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:16.306121355 +0000 UTC m=+2605.861749705" watchObservedRunningTime="2026-11-02 21:56:16.30911177 +0000 UTC m=+2605.864740109" Nov 02 21:56:16.729075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:16.729046 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:19.174836 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:19.174792 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:20.508117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:20.508055 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:24.991415 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:24.991378 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:25.110630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:25.110597 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:25.728738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:25.728705 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:25.746613 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:25.746586 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:26.319571 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:26.319541 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 21:56:29.137923 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:29.137878 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:29.586584 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:29.586548 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:34.181163 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:34.181129 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:38.691927 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:38.691874 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:40.615520 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:56:40.615479 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:56:43.199059 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:43.199021 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:47.665847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:47.665810 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:52.280531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:52.280486 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:54.461912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:54.461878 4441 scope.go:122] "RemoveContainer" containerID="8fb5aca7eea9dd8178b791f43ba1115312a0c97fdb177f45fd37cb260eedb070" Nov 02 21:56:54.473589 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:54.473559 4441 scope.go:122] "RemoveContainer" containerID="e55887c3ee590be664e8e683f633a9e4ba83528ebf389353b21a10c0db71b6b7" Nov 02 21:56:54.485700 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:54.485674 4441 scope.go:122] "RemoveContainer" containerID="cfb7dddf6dc70cd1287748df24038af1114f35ae1e37277dbe1c0a86c22d0a8e" Nov 02 21:56:54.497213 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:54.497193 4441 scope.go:122] "RemoveContainer" containerID="8d2cad65e8742d51dbf841dbb322ce848700efa3f3112e9042ff1a179949a9f7" Nov 02 21:56:55.372170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:55.372129 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:56.895813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:56.895772 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:57.022196 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:56:57.022166 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:01.022902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.022864 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh"] Nov 02 21:57:01.028937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.028917 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.061942 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.061908 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-dshm\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.062118 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.061992 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-home\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.062170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.062106 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-model-cache\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.062170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.062162 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tmp-dir\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.062294 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.062237 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tls-certs\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.062403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.062382 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b842\" (UniqueName: \"kubernetes.io/projected/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-kube-api-access-6b842\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.081942 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.081906 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh"] Nov 02 21:57:01.112488 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.112458 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:01.122883 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.122856 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bcb857\"/\"kube-root-ca.crt\"" Nov 02 21:57:01.123029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.122926 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bcb857\"/\"default-dockercfg-rsc9j\"" Nov 02 21:57:01.123029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.122924 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bcb857\"/\"openshift-service-ca.crt\"" Nov 02 21:57:01.125647 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.125624 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bcb857\"/\"test-model-bcb857-kserve-self-signed-certs\"" Nov 02 21:57:01.164176 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164146 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b842\" (UniqueName: \"kubernetes.io/projected/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-kube-api-access-6b842\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164257 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-dshm\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164289 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-home\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164336 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-model-cache\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164383 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tmp-dir\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164451 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164410 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tls-certs\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164815 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164790 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-model-cache\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164825 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-home\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.164890 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.164843 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tmp-dir\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.166540 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.166517 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-dshm\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.166819 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.166800 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tls-certs\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.221869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.221833 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b842\" (UniqueName: \"kubernetes.io/projected/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-kube-api-access-6b842\") pod \"test-model-bcb857-kserve-5d7f8b8989-bmbxh\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.341618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.341584 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:01.488354 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.488319 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:01.717130 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:57:01.717100 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cdcaf39_4003_4ee2_9886_a1f46cc1857a.slice/crio-5f392495f700fbe86741bc2056a3342df02ce47a3faf180ba8b45c7d0b1e05c1 WatchSource:0}: Error finding container 5f392495f700fbe86741bc2056a3342df02ce47a3faf180ba8b45c7d0b1e05c1: Status 404 returned error can't find the container with id 5f392495f700fbe86741bc2056a3342df02ce47a3faf180ba8b45c7d0b1e05c1 Nov 02 21:57:01.718344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:01.718320 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh"] Nov 02 21:57:02.620833 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:02.620793 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" event={"ID":"2cdcaf39-4003-4ee2-9886-a1f46cc1857a","Type":"ContainerStarted","Data":"0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4"} Nov 02 21:57:02.621286 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:02.620844 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" event={"ID":"2cdcaf39-4003-4ee2-9886-a1f46cc1857a","Type":"ContainerStarted","Data":"5f392495f700fbe86741bc2056a3342df02ce47a3faf180ba8b45c7d0b1e05c1"} Nov 02 21:57:02.731273 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:02.731215 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" podStartSLOduration=2.73119575 podStartE2EDuration="2.73119575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:02.727606109 +0000 UTC m=+2652.283234470" watchObservedRunningTime="2026-11-02 21:57:02.73119575 +0000 UTC m=+2652.286824089" Nov 02 21:57:03.342267 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:03.342237 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:04.250403 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:04.250360 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:04.371179 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:04.371145 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:05.468821 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:05.468789 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:06.975794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:06.975761 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:07.184222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:07.184182 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:07.315887 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:07.314356 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:07.471695 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:07.471655 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c"] Nov 02 21:57:07.518359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:07.518320 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d0946d-nf62c"] Nov 02 21:57:07.843201 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:07.843154 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:08.584978 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:08.584933 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e" path="/var/lib/kubelet/pods/4ca363a0-c3bc-4ca8-95b1-d9ecf3a6164e/volumes" Nov 02 21:57:10.616547 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:57:10.616504 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:57:11.342676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:11.342642 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:11.355617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:11.355589 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:11.712231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:11.712155 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 21:57:11.984520 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:11.984439 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:12.325232 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:12.325197 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc"] Nov 02 21:57:12.325461 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:12.325438 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" containerID="cri-o://0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41" gracePeriod=30 Nov 02 21:57:13.411431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:13.411397 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:13.510326 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:13.510287 4441 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:57:13.510504 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:13.510364 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:57:13.536911 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:13.536871 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:13.729799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:13.729718 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:14.530112 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.530078 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv"] Nov 02 21:57:14.534598 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.534579 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.589547 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.589517 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv"] Nov 02 21:57:14.630345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.630275 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:57:14.630345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.630322 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.630345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.630322 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.630633 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.630381 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:57:14.647669 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.647634 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.647847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.647826 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.647981 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.647953 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccnr\" (UniqueName: \"kubernetes.io/projected/d48e6772-a27b-488b-92d8-f37454bf7e70-kube-api-access-7ccnr\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.749329 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.749301 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.749494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.749388 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccnr\" (UniqueName: \"kubernetes.io/projected/d48e6772-a27b-488b-92d8-f37454bf7e70-kube-api-access-7ccnr\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.749494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.749437 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.750073 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.750053 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.751630 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.751611 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.802891 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.802833 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccnr\" (UniqueName: \"kubernetes.io/projected/d48e6772-a27b-488b-92d8-f37454bf7e70-kube-api-access-7ccnr\") pod \"maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:14.848343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:14.848322 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:15.224101 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:57:15.224070 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48e6772_a27b_488b_92d8_f37454bf7e70.slice/crio-1922931180cb4f4ae37bb9c5cbeb4ee2f14da067cf33ebb52cdd7bf4083f1218 WatchSource:0}: Error finding container 1922931180cb4f4ae37bb9c5cbeb4ee2f14da067cf33ebb52cdd7bf4083f1218: Status 404 returned error can't find the container with id 1922931180cb4f4ae37bb9c5cbeb4ee2f14da067cf33ebb52cdd7bf4083f1218 Nov 02 21:57:15.225713 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:15.225673 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv"] Nov 02 21:57:15.735611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:15.735511 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" event={"ID":"d48e6772-a27b-488b-92d8-f37454bf7e70","Type":"ContainerStarted","Data":"082426cc3d649e77441503edc969ab98044f2baff9239cc2debb57a1961b12a1"} Nov 02 21:57:15.735611 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:15.735586 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" event={"ID":"d48e6772-a27b-488b-92d8-f37454bf7e70","Type":"ContainerStarted","Data":"1922931180cb4f4ae37bb9c5cbeb4ee2f14da067cf33ebb52cdd7bf4083f1218"} Nov 02 21:57:15.842523 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:15.842458 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" podStartSLOduration=1.842444286 podStartE2EDuration="1.842444286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:15.842363549 +0000 UTC m=+2665.397991889" watchObservedRunningTime="2026-11-02 21:57:15.842444286 +0000 UTC m=+2665.398072622" Nov 02 21:57:16.746143 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:16.746110 4441 generic.go:356] "Generic (PLEG): container finished" podID="d48e6772-a27b-488b-92d8-f37454bf7e70" containerID="082426cc3d649e77441503edc969ab98044f2baff9239cc2debb57a1961b12a1" exitCode=0 Nov 02 21:57:16.746593 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:16.746192 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" event={"ID":"d48e6772-a27b-488b-92d8-f37454bf7e70","Type":"ContainerDied","Data":"082426cc3d649e77441503edc969ab98044f2baff9239cc2debb57a1961b12a1"} Nov 02 21:57:16.840250 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:16.840207 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:17.591484 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.591424 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:57:17.687639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.687603 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-envoy\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.687935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.687908 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-certs\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.687953 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0216113-4b24-436b-801e-2c29e3bfb3c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0216113-4b24-436b-801e-2c29e3bfb3c3-istiod-ca-cert\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688012 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-data\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688076 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-token\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-token\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688181 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688133 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-podinfo\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688182 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-socket\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688225 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-certs" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:57:17.688282 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688247 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-credential-socket\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688354 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-kube-api-access-psh4c\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-kube-api-access-psh4c\") pod \"d0216113-4b24-436b-801e-2c29e3bfb3c3\" (UID: \"d0216113-4b24-436b-801e-2c29e3bfb3c3\") " Nov 02 21:57:17.688735 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688591 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-socket" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:57:17.689058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688932 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.689058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.688958 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.689229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.689058 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-credential-socket" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:57:17.689229 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.689165 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-data" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:57:17.689455 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.689418 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0216113-4b24-436b-801e-2c29e3bfb3c3-istiod-ca-cert" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:57:17.690209 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.690183 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-envoy" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:57:17.690858 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.690837 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-token" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:17.691026 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.691003 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-podinfo" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:57:17.691157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.691138 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-kube-api-access-psh4c" pod "d0216113-4b24-436b-801e-2c29e3bfb3c3" (UID: "d0216113-4b24-436b-801e-2c29e3bfb3c3"). InnerVolumeSpecName "kube-api-access-psh4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:17.756773 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.756738 4441 generic.go:356] "Generic (PLEG): container finished" podID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerID="0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41" exitCode=0 Nov 02 21:57:17.757265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.756817 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" Nov 02 21:57:17.757265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.756816 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" event={"ID":"d0216113-4b24-436b-801e-2c29e3bfb3c3","Type":"ContainerDied","Data":"0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41"} Nov 02 21:57:17.757265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.756927 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc" event={"ID":"d0216113-4b24-436b-801e-2c29e3bfb3c3","Type":"ContainerDied","Data":"67a3a969590f0bdd6b41ecd9178a8e28800882ef40b902a6f6e583c2d381f085"} Nov 02 21:57:17.757265 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.757012 4441 scope.go:122] "RemoveContainer" containerID="0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41" Nov 02 21:57:17.774339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.774312 4441 scope.go:122] "RemoveContainer" containerID="18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02" Nov 02 21:57:17.789070 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.788961 4441 scope.go:122] "RemoveContainer" containerID="0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41" Nov 02 21:57:17.789605 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:57:17.789462 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41\": container with ID starting with 0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41 not found: ID does not exist" containerID="0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41" Nov 02 21:57:17.789605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.789504 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41"} err="failed to get container status \"0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41\": rpc error: code = NotFound desc = could not find container \"0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41\": container with ID starting with 0285410d6fe164ecb8461f7743a51e63e431d46f5f329da6f48c394da9365b41 not found: ID does not exist" Nov 02 21:57:17.789605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.789531 4441 scope.go:122] "RemoveContainer" containerID="18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02" Nov 02 21:57:17.790258 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.790225 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0216113-4b24-436b-801e-2c29e3bfb3c3-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.790356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.790261 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.790356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.790290 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.790356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.790305 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.790356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.790319 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.790356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.790332 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psh4c\" (UniqueName: \"kubernetes.io/projected/d0216113-4b24-436b-801e-2c29e3bfb3c3-kube-api-access-psh4c\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.790356 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.790346 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0216113-4b24-436b-801e-2c29e3bfb3c3-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.791089 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:57:17.791063 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02\": container with ID starting with 18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02 not found: ID does not exist" containerID="18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02" Nov 02 21:57:17.791188 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.791098 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02"} err="failed to get container status \"18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02\": rpc error: code = NotFound desc = could not find container \"18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02\": container with ID starting with 18f6cf3326cec7d6592b6041694c98eecce21245817e62ebf9cc766e4b598f02 not found: ID does not exist" Nov 02 21:57:17.919561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.919528 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc"] Nov 02 21:57:17.923637 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.923605 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-bbrxc"] Nov 02 21:57:17.928494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.928476 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:17.993233 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.993203 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-openshift-ca\") pod \"d48e6772-a27b-488b-92d8-f37454bf7e70\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " Nov 02 21:57:17.993392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.993316 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48e6772-a27b-488b-92d8-f37454bf7e70-kube-api-access-7ccnr\" (UniqueName: \"kubernetes.io/projected/d48e6772-a27b-488b-92d8-f37454bf7e70-kube-api-access-7ccnr\") pod \"d48e6772-a27b-488b-92d8-f37454bf7e70\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " Nov 02 21:57:17.993392 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.993360 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-xks-ca\") pod \"d48e6772-a27b-488b-92d8-f37454bf7e70\" (UID: \"d48e6772-a27b-488b-92d8-f37454bf7e70\") " Nov 02 21:57:17.993648 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.993624 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-openshift-ca" pod "d48e6772-a27b-488b-92d8-f37454bf7e70" (UID: "d48e6772-a27b-488b-92d8-f37454bf7e70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:57:17.993938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.993912 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:17.995395 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.995373 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48e6772-a27b-488b-92d8-f37454bf7e70-kube-api-access-7ccnr" pod "d48e6772-a27b-488b-92d8-f37454bf7e70" (UID: "d48e6772-a27b-488b-92d8-f37454bf7e70"). InnerVolumeSpecName "kube-api-access-7ccnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:17.995497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:17.995411 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-xks-ca" pod "d48e6772-a27b-488b-92d8-f37454bf7e70" (UID: "d48e6772-a27b-488b-92d8-f37454bf7e70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:18.095042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:18.095005 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ccnr\" (UniqueName: \"kubernetes.io/projected/d48e6772-a27b-488b-92d8-f37454bf7e70-kube-api-access-7ccnr\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:18.095042 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:18.095038 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48e6772-a27b-488b-92d8-f37454bf7e70-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:18.585307 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:18.585279 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" path="/var/lib/kubelet/pods/d0216113-4b24-436b-801e-2c29e3bfb3c3/volumes" Nov 02 21:57:18.765939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:18.765849 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" Nov 02 21:57:18.765939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:18.765861 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv" event={"ID":"d48e6772-a27b-488b-92d8-f37454bf7e70","Type":"ContainerDied","Data":"1922931180cb4f4ae37bb9c5cbeb4ee2f14da067cf33ebb52cdd7bf4083f1218"} Nov 02 21:57:18.765939 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:18.765915 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1922931180cb4f4ae37bb9c5cbeb4ee2f14da067cf33ebb52cdd7bf4083f1218" Nov 02 21:57:21.597119 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:21.597064 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:26.037047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:26.037013 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf"] Nov 02 21:57:26.084406 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:26.084379 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:26.275545 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:26.275512 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:26.416866 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:26.416836 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:30.607487 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:30.607450 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:31.038087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.037951 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" podUID="45042402-da10-4f5d-8bfd-05b3e36ba5b4" containerName="payload-pre-processing" containerID="cri-o://1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3" gracePeriod=25 Nov 02 21:57:31.328555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.328526 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:57:31.500466 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.500411 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45042402-da10-4f5d-8bfd-05b3e36ba5b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45042402-da10-4f5d-8bfd-05b3e36ba5b4-plugins-config-volume\") pod \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " Nov 02 21:57:31.500674 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.500514 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45042402-da10-4f5d-8bfd-05b3e36ba5b4-kube-api-access-z5mrw\" (UniqueName: \"kubernetes.io/projected/45042402-da10-4f5d-8bfd-05b3e36ba5b4-kube-api-access-z5mrw\") pod \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\" (UID: \"45042402-da10-4f5d-8bfd-05b3e36ba5b4\") " Nov 02 21:57:31.500868 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.500843 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45042402-da10-4f5d-8bfd-05b3e36ba5b4-plugins-config-volume" pod "45042402-da10-4f5d-8bfd-05b3e36ba5b4" (UID: "45042402-da10-4f5d-8bfd-05b3e36ba5b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:57:31.501090 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.501071 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45042402-da10-4f5d-8bfd-05b3e36ba5b4-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:31.502624 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.502600 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45042402-da10-4f5d-8bfd-05b3e36ba5b4-kube-api-access-z5mrw" pod "45042402-da10-4f5d-8bfd-05b3e36ba5b4" (UID: "45042402-da10-4f5d-8bfd-05b3e36ba5b4"). InnerVolumeSpecName "kube-api-access-z5mrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:31.602264 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.602234 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mrw\" (UniqueName: \"kubernetes.io/projected/45042402-da10-4f5d-8bfd-05b3e36ba5b4-kube-api-access-z5mrw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:31.881915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.881833 4441 generic.go:356] "Generic (PLEG): container finished" podID="45042402-da10-4f5d-8bfd-05b3e36ba5b4" containerID="1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3" exitCode=0 Nov 02 21:57:31.881915 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.881885 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" Nov 02 21:57:31.882457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.881914 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" event={"ID":"45042402-da10-4f5d-8bfd-05b3e36ba5b4","Type":"ContainerDied","Data":"1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3"} Nov 02 21:57:31.882457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.881998 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf" event={"ID":"45042402-da10-4f5d-8bfd-05b3e36ba5b4","Type":"ContainerDied","Data":"09940b307521f94341a8cb616d72e95baa23d2db5354398befd996091f089eca"} Nov 02 21:57:31.882457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.882030 4441 scope.go:122] "RemoveContainer" containerID="1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3" Nov 02 21:57:31.893340 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.893316 4441 scope.go:122] "RemoveContainer" containerID="1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3" Nov 02 21:57:31.893643 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:57:31.893622 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3\": container with ID starting with 1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3 not found: ID does not exist" containerID="1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3" Nov 02 21:57:31.893741 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:31.893648 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3"} err="failed to get container status \"1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3\": rpc error: code = NotFound desc = could not find container \"1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3\": container with ID starting with 1a565a2dcc173363125850e1f12ce0f37ffd7c7b71fe75aaf435966247e116e3 not found: ID does not exist" Nov 02 21:57:32.039472 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:32.039440 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf"] Nov 02 21:57:32.086538 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:32.086510 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d2b29-7d6d7767-724mf"] Nov 02 21:57:32.589827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:32.589778 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45042402-da10-4f5d-8bfd-05b3e36ba5b4" path="/var/lib/kubelet/pods/45042402-da10-4f5d-8bfd-05b3e36ba5b4/volumes" Nov 02 21:57:35.091440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:35.091405 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:39.581022 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:39.580952 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:40.618871 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:57:40.618832 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:57:44.094096 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:44.094059 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:48.545857 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:48.545816 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:51.596270 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:51.596237 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:57:51.597418 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:51.597393 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 21:57:51.605279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:51.605259 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:57:51.606420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:51.606404 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 21:57:51.618318 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:51.618293 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:57:51.620003 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:51.619983 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 21:57:53.113539 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:53.113504 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:55.705482 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:55.705438 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:57.608582 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:57.608532 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:57.756154 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:57.756117 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7"] Nov 02 21:57:57.756377 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:57.756302 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" podUID="f9949816-0828-42b1-831a-27f618456cf1" containerName="discovery" containerID="cri-o://5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432" gracePeriod=30 Nov 02 21:57:57.942420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:57.942382 4441 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-m56v7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:8080/ready\": dial tcp 10.133.0.157:8080: connect: connection refused" start-of-body= Nov 02 21:57:57.942607 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:57.942440 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" podUID="f9949816-0828-42b1-831a-27f618456cf1" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.157:8080/ready\": dial tcp 10.133.0.157:8080: connect: connection refused" Nov 02 21:57:58.051435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.051408 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:57:58.084508 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.084473 4441 generic.go:356] "Generic (PLEG): container finished" podID="f9949816-0828-42b1-831a-27f618456cf1" containerID="5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432" exitCode=0 Nov 02 21:57:58.084784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.084533 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" Nov 02 21:57:58.084913 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.084549 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" event={"ID":"f9949816-0828-42b1-831a-27f618456cf1","Type":"ContainerDied","Data":"5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432"} Nov 02 21:57:58.084981 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.084908 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7" event={"ID":"f9949816-0828-42b1-831a-27f618456cf1","Type":"ContainerDied","Data":"4c17678124d764fd94e67cbe386ae4bb2d3f61d24599a58acbc1ad02ec1e452c"} Nov 02 21:57:58.084981 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.084939 4441 scope.go:122] "RemoveContainer" containerID="5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432" Nov 02 21:57:58.098368 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.098347 4441 scope.go:122] "RemoveContainer" containerID="5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432" Nov 02 21:57:58.099491 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:57:58.099425 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432\": container with ID starting with 5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432 not found: ID does not exist" containerID="5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432" Nov 02 21:57:58.099491 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.099469 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432"} err="failed to get container status \"5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432\": rpc error: code = NotFound desc = could not find container \"5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432\": container with ID starting with 5a4485a711f144848b0442efcaadfd0513fb5182fba2a1a96fd6db833b7e6432 not found: ID does not exist" Nov 02 21:57:58.134075 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134006 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-dns-cert\") pod \"f9949816-0828-42b1-831a-27f618456cf1\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " Nov 02 21:57:58.134198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134090 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9949816-0828-42b1-831a-27f618456cf1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9949816-0828-42b1-831a-27f618456cf1-local-certs\") pod \"f9949816-0828-42b1-831a-27f618456cf1\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " Nov 02 21:57:58.134198 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134136 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-kubeconfig\") pod \"f9949816-0828-42b1-831a-27f618456cf1\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " Nov 02 21:57:58.134278 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134231 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-cacerts\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-cacerts\") pod \"f9949816-0828-42b1-831a-27f618456cf1\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " Nov 02 21:57:58.134313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134296 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-kube-api-access-plch2\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-kube-api-access-plch2\") pod \"f9949816-0828-42b1-831a-27f618456cf1\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " Nov 02 21:57:58.134359 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134344 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-ca-configmap\") pod \"f9949816-0828-42b1-831a-27f618456cf1\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " Nov 02 21:57:58.134404 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134375 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-istio-token\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-istio-token\") pod \"f9949816-0828-42b1-831a-27f618456cf1\" (UID: \"f9949816-0828-42b1-831a-27f618456cf1\") " Nov 02 21:57:58.134828 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.134757 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-ca-configmap" pod "f9949816-0828-42b1-831a-27f618456cf1" (UID: "f9949816-0828-42b1-831a-27f618456cf1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:57:58.135060 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.135040 4441 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-ca-configmap\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:58.137068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.136946 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-dns-cert" pod "f9949816-0828-42b1-831a-27f618456cf1" (UID: "f9949816-0828-42b1-831a-27f618456cf1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:58.137068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.137020 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-cacerts" pod "f9949816-0828-42b1-831a-27f618456cf1" (UID: "f9949816-0828-42b1-831a-27f618456cf1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:58.137068 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.137057 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-kubeconfig" pod "f9949816-0828-42b1-831a-27f618456cf1" (UID: "f9949816-0828-42b1-831a-27f618456cf1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:58.137291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.137096 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-kube-api-access-plch2" pod "f9949816-0828-42b1-831a-27f618456cf1" (UID: "f9949816-0828-42b1-831a-27f618456cf1"). InnerVolumeSpecName "kube-api-access-plch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:58.137291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.137129 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-istio-token" pod "f9949816-0828-42b1-831a-27f618456cf1" (UID: "f9949816-0828-42b1-831a-27f618456cf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:58.137291 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.137170 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9949816-0828-42b1-831a-27f618456cf1-local-certs" pod "f9949816-0828-42b1-831a-27f618456cf1" (UID: "f9949816-0828-42b1-831a-27f618456cf1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:57:58.236756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.236719 4441 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-csr-dns-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:58.236756 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.236754 4441 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9949816-0828-42b1-831a-27f618456cf1-local-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:58.236955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.236765 4441 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-istio-kubeconfig\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:58.236955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.236773 4441 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9949816-0828-42b1-831a-27f618456cf1-cacerts\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:58.236955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.236783 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plch2\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-kube-api-access-plch2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:58.236955 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.236792 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9949816-0828-42b1-831a-27f618456cf1-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:58.503478 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.503442 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7"] Nov 02 21:57:58.507151 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.507121 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m56v7"] Nov 02 21:57:58.587171 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:57:58.587138 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9949816-0828-42b1-831a-27f618456cf1" path="/var/lib/kubelet/pods/f9949816-0828-42b1-831a-27f618456cf1/volumes" Nov 02 21:58:02.177605 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:02.177567 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:06.701730 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:06.701696 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:10.618723 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:58:10.618677 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:58:11.307642 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:11.307606 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:15.869168 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:15.869109 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:20.327856 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:20.327826 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:22.560899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:22.560863 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:23.404755 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:23.404717 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:23.528314 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:23.528278 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:24.801636 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:24.801603 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:29.383420 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.383384 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:29.530142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530111 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4"] Nov 02 21:58:29.530837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530807 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" Nov 02 21:58:29.530837 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530831 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530851 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d48e6772-a27b-488b-92d8-f37454bf7e70" containerName="revoke-keys" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530859 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48e6772-a27b-488b-92d8-f37454bf7e70" containerName="revoke-keys" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530892 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f9949816-0828-42b1-831a-27f618456cf1" containerName="discovery" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530900 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9949816-0828-42b1-831a-27f618456cf1" containerName="discovery" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530929 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530936 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530944 4441 cpu_manager.go:414] "containerMap: removing container" podUID="45042402-da10-4f5d-8bfd-05b3e36ba5b4" containerName="payload-pre-processing" Nov 02 21:58:29.531076 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.530951 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45042402-da10-4f5d-8bfd-05b3e36ba5b4" containerName="payload-pre-processing" Nov 02 21:58:29.531494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.531141 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" Nov 02 21:58:29.531494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.531155 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48e6772-a27b-488b-92d8-f37454bf7e70" containerName="revoke-keys" Nov 02 21:58:29.531494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.531170 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9949816-0828-42b1-831a-27f618456cf1" containerName="discovery" Nov 02 21:58:29.531494 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.531184 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="45042402-da10-4f5d-8bfd-05b3e36ba5b4" containerName="payload-pre-processing" Nov 02 21:58:29.534661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.534640 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.587464 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.587432 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4"] Nov 02 21:58:29.601705 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.601679 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnkz\" (UniqueName: \"kubernetes.io/projected/1180da75-b7a5-4a10-a580-430af8675ac4-kube-api-access-6rnkz\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.601840 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.601762 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.601901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.601873 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.629235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.629213 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:58:29.629235 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.629220 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:58:29.629402 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.629281 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:58:29.631217 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.631199 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:58:29.703364 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.703300 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnkz\" (UniqueName: \"kubernetes.io/projected/1180da75-b7a5-4a10-a580-430af8675ac4-kube-api-access-6rnkz\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.703498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.703370 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.703498 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.703404 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.704295 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.704269 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.705944 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.705924 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.757129 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.757102 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnkz\" (UniqueName: \"kubernetes.io/projected/1180da75-b7a5-4a10-a580-430af8675ac4-kube-api-access-6rnkz\") pod \"maas-api-revoke-sub-410f0750-zr5m4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:29.849025 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:29.848996 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:30.220880 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:58:30.220849 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1180da75_b7a5_4a10_a580_430af8675ac4.slice/crio-a13af8c93927b309dda5dded98287aee4155fac890856b0c5a9f231d939c1f91 WatchSource:0}: Error finding container a13af8c93927b309dda5dded98287aee4155fac890856b0c5a9f231d939c1f91: Status 404 returned error can't find the container with id a13af8c93927b309dda5dded98287aee4155fac890856b0c5a9f231d939c1f91 Nov 02 21:58:30.222779 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:30.222747 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4"] Nov 02 21:58:30.344124 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:30.344088 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" event={"ID":"1180da75-b7a5-4a10-a580-430af8675ac4","Type":"ContainerStarted","Data":"a13af8c93927b309dda5dded98287aee4155fac890856b0c5a9f231d939c1f91"} Nov 02 21:58:31.048272 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:31.048241 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz"] Nov 02 21:58:31.092554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:31.092518 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be55312-gtvcz"] Nov 02 21:58:31.353825 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:31.353791 4441 generic.go:356] "Generic (PLEG): container finished" podID="1180da75-b7a5-4a10-a580-430af8675ac4" containerID="f58c92d1868244a41c8441b6dbbbe84898e7e6341d8a3567ca9d39fe6e6a3cf5" exitCode=0 Nov 02 21:58:31.354040 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:31.353884 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" event={"ID":"1180da75-b7a5-4a10-a580-430af8675ac4","Type":"ContainerDied","Data":"f58c92d1868244a41c8441b6dbbbe84898e7e6341d8a3567ca9d39fe6e6a3cf5"} Nov 02 21:58:32.520323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.520291 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:32.585541 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.585507 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07340c30-44f9-467a-a590-30b8cca7b737" path="/var/lib/kubelet/pods/07340c30-44f9-467a-a590-30b8cca7b737/volumes" Nov 02 21:58:32.638387 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.638315 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-openshift-ca\") pod \"1180da75-b7a5-4a10-a580-430af8675ac4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " Nov 02 21:58:32.638554 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.638537 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1180da75-b7a5-4a10-a580-430af8675ac4-kube-api-access-6rnkz\" (UniqueName: \"kubernetes.io/projected/1180da75-b7a5-4a10-a580-430af8675ac4-kube-api-access-6rnkz\") pod \"1180da75-b7a5-4a10-a580-430af8675ac4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " Nov 02 21:58:32.638643 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.638628 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-xks-ca\") pod \"1180da75-b7a5-4a10-a580-430af8675ac4\" (UID: \"1180da75-b7a5-4a10-a580-430af8675ac4\") " Nov 02 21:58:32.639661 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.639632 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-openshift-ca" pod "1180da75-b7a5-4a10-a580-430af8675ac4" (UID: "1180da75-b7a5-4a10-a580-430af8675ac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:32.642832 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.642805 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1180da75-b7a5-4a10-a580-430af8675ac4-kube-api-access-6rnkz" pod "1180da75-b7a5-4a10-a580-430af8675ac4" (UID: "1180da75-b7a5-4a10-a580-430af8675ac4"). InnerVolumeSpecName "kube-api-access-6rnkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:32.642960 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.642882 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-xks-ca" pod "1180da75-b7a5-4a10-a580-430af8675ac4" (UID: "1180da75-b7a5-4a10-a580-430af8675ac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:32.740416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.740387 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnkz\" (UniqueName: \"kubernetes.io/projected/1180da75-b7a5-4a10-a580-430af8675ac4-kube-api-access-6rnkz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:32.740416 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.740418 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:32.740639 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:32.740433 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1180da75-b7a5-4a10-a580-430af8675ac4-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:33.370653 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:33.370622 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" Nov 02 21:58:33.370830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:33.370651 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4" event={"ID":"1180da75-b7a5-4a10-a580-430af8675ac4","Type":"ContainerDied","Data":"a13af8c93927b309dda5dded98287aee4155fac890856b0c5a9f231d939c1f91"} Nov 02 21:58:33.370830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:33.370699 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a13af8c93927b309dda5dded98287aee4155fac890856b0c5a9f231d939c1f91" Nov 02 21:58:33.898313 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:33.898270 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:38.308526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:38.308471 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:40.619806 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:58:40.619761 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:58:41.438531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:41.438498 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:42.375632 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.375599 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q"] Nov 02 21:58:42.376350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.376325 4441 cpu_manager.go:414] "containerMap: removing container" podUID="1180da75-b7a5-4a10-a580-430af8675ac4" containerName="revoke-keys" Nov 02 21:58:42.376350 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.376347 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1180da75-b7a5-4a10-a580-430af8675ac4" containerName="revoke-keys" Nov 02 21:58:42.376524 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.376505 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0216113-4b24-436b-801e-2c29e3bfb3c3" containerName="istio-proxy" Nov 02 21:58:42.376586 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.376529 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="1180da75-b7a5-4a10-a580-430af8675ac4" containerName="revoke-keys" Nov 02 21:58:42.380113 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.380094 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.432738 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.432705 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q"] Nov 02 21:58:42.456592 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.456554 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814422d2-590b-49e4-8da2-d92333945435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.456762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.456606 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814422d2-590b-49e4-8da2-d92333945435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.456762 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.456623 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfngz\" (UniqueName: \"kubernetes.io/projected/814422d2-590b-49e4-8da2-d92333945435-kube-api-access-xfngz\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.474827 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.474795 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:58:42.474983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.474842 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:58:42.474983 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.474896 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:58:42.476561 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.476543 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:58:42.558088 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.558043 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814422d2-590b-49e4-8da2-d92333945435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.558289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.558109 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814422d2-590b-49e4-8da2-d92333945435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.558289 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.558139 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfngz\" (UniqueName: \"kubernetes.io/projected/814422d2-590b-49e4-8da2-d92333945435-kube-api-access-xfngz\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.558881 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.558859 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814422d2-590b-49e4-8da2-d92333945435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.560980 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.560938 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814422d2-590b-49e4-8da2-d92333945435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.612486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.612453 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfngz\" (UniqueName: \"kubernetes.io/projected/814422d2-590b-49e4-8da2-d92333945435-kube-api-access-xfngz\") pod \"maas-api-revoke-sub-4810d2a3-hsh7q\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.694821 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.694743 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:42.808603 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:42.808556 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:43.082177 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:43.082143 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q"] Nov 02 21:58:43.083764 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:58:43.083722 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814422d2_590b_49e4_8da2_d92333945435.slice/crio-0a9665c3741263e842bfdc79f95184697dfa18b6accff4380a348f7ab8329482 WatchSource:0}: Error finding container 0a9665c3741263e842bfdc79f95184697dfa18b6accff4380a348f7ab8329482: Status 404 returned error can't find the container with id 0a9665c3741263e842bfdc79f95184697dfa18b6accff4380a348f7ab8329482 Nov 02 21:58:43.456732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:43.456693 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" event={"ID":"814422d2-590b-49e4-8da2-d92333945435","Type":"ContainerStarted","Data":"36a8b2a9000a4f14cf2a7b44a5ea4f2ff7d73f073716adb7e5231ef14d3e2985"} Nov 02 21:58:43.457249 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:43.456751 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" event={"ID":"814422d2-590b-49e4-8da2-d92333945435","Type":"ContainerStarted","Data":"0a9665c3741263e842bfdc79f95184697dfa18b6accff4380a348f7ab8329482"} Nov 02 21:58:43.569867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:43.569795 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" podStartSLOduration=1.5697764520000002 podStartE2EDuration="1.569776452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:43.565458011 +0000 UTC m=+2753.121086343" watchObservedRunningTime="2026-11-02 21:58:43.569776452 +0000 UTC m=+2753.125404883" Nov 02 21:58:44.403280 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:44.403245 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:44.466158 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:44.466126 4441 generic.go:356] "Generic (PLEG): container finished" podID="814422d2-590b-49e4-8da2-d92333945435" containerID="36a8b2a9000a4f14cf2a7b44a5ea4f2ff7d73f073716adb7e5231ef14d3e2985" exitCode=0 Nov 02 21:58:44.466537 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:44.466178 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" event={"ID":"814422d2-590b-49e4-8da2-d92333945435","Type":"ContainerDied","Data":"36a8b2a9000a4f14cf2a7b44a5ea4f2ff7d73f073716adb7e5231ef14d3e2985"} Nov 02 21:58:45.630134 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.630105 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:45.696066 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.696030 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814422d2-590b-49e4-8da2-d92333945435-kube-api-access-xfngz\" (UniqueName: \"kubernetes.io/projected/814422d2-590b-49e4-8da2-d92333945435-kube-api-access-xfngz\") pod \"814422d2-590b-49e4-8da2-d92333945435\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " Nov 02 21:58:45.696247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.696112 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814422d2-590b-49e4-8da2-d92333945435-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814422d2-590b-49e4-8da2-d92333945435-maas-api-xks-ca\") pod \"814422d2-590b-49e4-8da2-d92333945435\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " Nov 02 21:58:45.696247 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.696145 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814422d2-590b-49e4-8da2-d92333945435-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814422d2-590b-49e4-8da2-d92333945435-maas-api-openshift-ca\") pod \"814422d2-590b-49e4-8da2-d92333945435\" (UID: \"814422d2-590b-49e4-8da2-d92333945435\") " Nov 02 21:58:45.696506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.696479 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814422d2-590b-49e4-8da2-d92333945435-maas-api-openshift-ca" pod "814422d2-590b-49e4-8da2-d92333945435" (UID: "814422d2-590b-49e4-8da2-d92333945435"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:45.696794 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.696769 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814422d2-590b-49e4-8da2-d92333945435-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:45.698323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.698297 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814422d2-590b-49e4-8da2-d92333945435-kube-api-access-xfngz" pod "814422d2-590b-49e4-8da2-d92333945435" (UID: "814422d2-590b-49e4-8da2-d92333945435"). InnerVolumeSpecName "kube-api-access-xfngz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:45.698595 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.698572 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814422d2-590b-49e4-8da2-d92333945435-maas-api-xks-ca" pod "814422d2-590b-49e4-8da2-d92333945435" (UID: "814422d2-590b-49e4-8da2-d92333945435"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:45.797948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.797870 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfngz\" (UniqueName: \"kubernetes.io/projected/814422d2-590b-49e4-8da2-d92333945435-kube-api-access-xfngz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:45.797948 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:45.797905 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814422d2-590b-49e4-8da2-d92333945435-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.484194 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:46.484165 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" Nov 02 21:58:46.484372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:46.484165 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q" event={"ID":"814422d2-590b-49e4-8da2-d92333945435","Type":"ContainerDied","Data":"0a9665c3741263e842bfdc79f95184697dfa18b6accff4380a348f7ab8329482"} Nov 02 21:58:46.484372 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:46.484309 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9665c3741263e842bfdc79f95184697dfa18b6accff4380a348f7ab8329482" Nov 02 21:58:47.306257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:47.306225 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:51.790083 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:51.790031 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:55.400047 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:55.400012 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:55.526243 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:55.526192 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:56.050796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:56.050766 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:56.293343 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:58:56.293308 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:00.792290 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:00.792248 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:05.311300 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:05.311249 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:09.766784 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:09.766753 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:10.616271 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:59:10.616235 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:59:14.237581 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:14.237545 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:18.729760 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:18.729701 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:19.738142 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:19.738113 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:20.267344 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.267302 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9"] Nov 02 21:59:20.267946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.267928 4441 cpu_manager.go:414] "containerMap: removing container" podUID="814422d2-590b-49e4-8da2-d92333945435" containerName="revoke-keys" Nov 02 21:59:20.267946 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.267947 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814422d2-590b-49e4-8da2-d92333945435" containerName="revoke-keys" Nov 02 21:59:20.268126 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.268073 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="814422d2-590b-49e4-8da2-d92333945435" containerName="revoke-keys" Nov 02 21:59:20.271486 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.271458 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.311919 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.311876 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpzj\" (UniqueName: \"kubernetes.io/projected/0df88869-0b53-4e2f-8974-dcb2d5b43468-kube-api-access-vtpzj\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.312157 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.312015 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.312231 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.312165 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.322574 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.322537 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9"] Nov 02 21:59:20.365274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.365213 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:59:20.365274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.365212 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:59:20.365274 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.365212 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:59:20.368396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.368373 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:59:20.413199 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.413164 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpzj\" (UniqueName: \"kubernetes.io/projected/0df88869-0b53-4e2f-8974-dcb2d5b43468-kube-api-access-vtpzj\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.413388 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.413257 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.413463 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.413408 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.414139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.414091 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.415672 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.415650 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.466916 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.466887 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpzj\" (UniqueName: \"kubernetes.io/projected/0df88869-0b53-4e2f-8974-dcb2d5b43468-kube-api-access-vtpzj\") pod \"maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.585717 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.585678 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:20.981315 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:59:20.980870 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df88869_0b53_4e2f_8974_dcb2d5b43468.slice/crio-79e6358f0449d3f413a2cfc13b8482cbb8f8daab5382e779dce15550849baadc WatchSource:0}: Error finding container 79e6358f0449d3f413a2cfc13b8482cbb8f8daab5382e779dce15550849baadc: Status 404 returned error can't find the container with id 79e6358f0449d3f413a2cfc13b8482cbb8f8daab5382e779dce15550849baadc Nov 02 21:59:20.981849 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:20.981617 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9"] Nov 02 21:59:21.831058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:21.831018 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" event={"ID":"0df88869-0b53-4e2f-8974-dcb2d5b43468","Type":"ContainerStarted","Data":"8adbb6fb5861eea86c735ae68a175bed0443a600784a0787a7d8336cb8c960e8"} Nov 02 21:59:21.831246 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:21.831089 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" event={"ID":"0df88869-0b53-4e2f-8974-dcb2d5b43468","Type":"ContainerStarted","Data":"79e6358f0449d3f413a2cfc13b8482cbb8f8daab5382e779dce15550849baadc"} Nov 02 21:59:21.940382 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:21.940329 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" podStartSLOduration=1.940314172 podStartE2EDuration="1.940314172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:21.937519636 +0000 UTC m=+2791.493147973" watchObservedRunningTime="2026-11-02 21:59:21.940314172 +0000 UTC m=+2791.495942555" Nov 02 21:59:22.839842 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:22.839805 4441 generic.go:356] "Generic (PLEG): container finished" podID="0df88869-0b53-4e2f-8974-dcb2d5b43468" containerID="8adbb6fb5861eea86c735ae68a175bed0443a600784a0787a7d8336cb8c960e8" exitCode=0 Nov 02 21:59:22.840257 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:22.839875 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" event={"ID":"0df88869-0b53-4e2f-8974-dcb2d5b43468","Type":"ContainerDied","Data":"8adbb6fb5861eea86c735ae68a175bed0443a600784a0787a7d8336cb8c960e8"} Nov 02 21:59:23.763049 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:23.763013 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:24.019358 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.019301 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:24.087236 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.087211 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df88869-0b53-4e2f-8974-dcb2d5b43468-kube-api-access-vtpzj\" (UniqueName: \"kubernetes.io/projected/0df88869-0b53-4e2f-8974-dcb2d5b43468-kube-api-access-vtpzj\") pod \"0df88869-0b53-4e2f-8974-dcb2d5b43468\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " Nov 02 21:59:24.087393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.087252 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-openshift-ca\") pod \"0df88869-0b53-4e2f-8974-dcb2d5b43468\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " Nov 02 21:59:24.087393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.087341 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-xks-ca\") pod \"0df88869-0b53-4e2f-8974-dcb2d5b43468\" (UID: \"0df88869-0b53-4e2f-8974-dcb2d5b43468\") " Nov 02 21:59:24.087712 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.087682 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-openshift-ca" pod "0df88869-0b53-4e2f-8974-dcb2d5b43468" (UID: "0df88869-0b53-4e2f-8974-dcb2d5b43468"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:24.088000 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.087961 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:24.089473 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.089445 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df88869-0b53-4e2f-8974-dcb2d5b43468-kube-api-access-vtpzj" pod "0df88869-0b53-4e2f-8974-dcb2d5b43468" (UID: "0df88869-0b53-4e2f-8974-dcb2d5b43468"). InnerVolumeSpecName "kube-api-access-vtpzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:24.089909 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.089884 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-xks-ca" pod "0df88869-0b53-4e2f-8974-dcb2d5b43468" (UID: "0df88869-0b53-4e2f-8974-dcb2d5b43468"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:24.189284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.189248 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df88869-0b53-4e2f-8974-dcb2d5b43468-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:24.189284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.189286 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtpzj\" (UniqueName: \"kubernetes.io/projected/0df88869-0b53-4e2f-8974-dcb2d5b43468-kube-api-access-vtpzj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:24.859123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.859044 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" Nov 02 21:59:24.859123 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.859085 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9" event={"ID":"0df88869-0b53-4e2f-8974-dcb2d5b43468","Type":"ContainerDied","Data":"79e6358f0449d3f413a2cfc13b8482cbb8f8daab5382e779dce15550849baadc"} Nov 02 21:59:24.859360 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:24.859133 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e6358f0449d3f413a2cfc13b8482cbb8f8daab5382e779dce15550849baadc" Nov 02 21:59:27.878200 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:27.878172 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:27.998121 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:27.998087 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:28.589502 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:28.589462 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:32.007401 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:32.007372 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:32.125393 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:32.125352 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:33.550928 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:33.550785 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:38.481238 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:38.481200 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:40.618204 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:59:40.618162 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 21:59:41.046555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:41.046462 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs"] Nov 02 21:59:41.093304 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:41.093258 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk"] Nov 02 21:59:41.097901 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:41.097843 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18988557-cznvs"] Nov 02 21:59:41.101905 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:41.101875 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310b7510-gq7vk"] Nov 02 21:59:42.586405 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:42.586370 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841ecdc8-337b-4e93-84cf-606cfb161a88" path="/var/lib/kubelet/pods/841ecdc8-337b-4e93-84cf-606cfb161a88/volumes" Nov 02 21:59:42.586912 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:42.586808 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c27d6a-53df-4736-a77e-b59b60c828e6" path="/var/lib/kubelet/pods/e6c27d6a-53df-4736-a77e-b59b60c828e6/volumes" Nov 02 21:59:42.937782 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:42.937696 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:46.874922 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:46.874890 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:48.323771 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:48.323734 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:48.522506 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:48.522477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:48.645325 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:48.645236 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:48.911792 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:48.911701 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv"] Nov 02 21:59:48.958497 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:48.958461 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d2b29-xctpv"] Nov 02 21:59:49.284440 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:49.284352 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:50.589371 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:50.589321 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48e6772-a27b-488b-92d8-f37454bf7e70" path="/var/lib/kubelet/pods/d48e6772-a27b-488b-92d8-f37454bf7e70/volumes" Nov 02 21:59:52.500617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.500580 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g"] Nov 02 21:59:52.501161 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.500817 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" containerID="cri-o://138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9" gracePeriod=30 Nov 02 21:59:52.547640 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.547609 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2"] Nov 02 21:59:52.547863 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.547832 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" containerID="cri-o://a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495" gracePeriod=30 Nov 02 21:59:52.551396 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.551370 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr"] Nov 02 21:59:52.551564 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.551547 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" containerID="cri-o://4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee" gracePeriod=30 Nov 02 21:59:52.595336 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.595307 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj"] Nov 02 21:59:52.595518 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:52.595484 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" containerID="cri-o://889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48" gracePeriod=30 Nov 02 21:59:53.312222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:53.312186 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:54.606668 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:54.606635 4441 scope.go:122] "RemoveContainer" containerID="448e63af1ca2e9b354f0546cb7513fbd1ddd606035667727ddcd6d9373982824" Nov 02 21:59:54.618783 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:54.618752 4441 scope.go:122] "RemoveContainer" containerID="612477f45b5d7cfcc32275ec1166ca34ee2bbb120fff15eafdc4def5ef53744e" Nov 02 21:59:55.004807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:55.004692 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:55.136567 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:55.136504 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:55.318435 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:55.318402 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:56.092885 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.092816 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:56.392621 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.392525 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:56.612279 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.612251 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7"] Nov 02 21:59:56.613007 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.612984 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0df88869-0b53-4e2f-8974-dcb2d5b43468" containerName="revoke-keys" Nov 02 21:59:56.613087 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.613010 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df88869-0b53-4e2f-8974-dcb2d5b43468" containerName="revoke-keys" Nov 02 21:59:56.613169 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.613158 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df88869-0b53-4e2f-8974-dcb2d5b43468" containerName="revoke-keys" Nov 02 21:59:56.618222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.618201 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.673014 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.672923 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c4a6b9-openshift-default-dockercfg-g5zwv\"" Nov 02 21:59:56.692997 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.692939 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:56.705577 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705550 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-envoy\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705610 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-certs\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705634 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-credential-socket\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705687 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-podinfo\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705728 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705703 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-token\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705792 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-data\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705815 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbsx\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-kube-api-access-lzbsx\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705851 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-socket\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.705959 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.705890 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istiod-ca-cert\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.721411 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.720881 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7"] Nov 02 21:59:56.814618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.813755 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-data\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.814618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.814414 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-data\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.814618 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.814494 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbsx\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-kube-api-access-lzbsx\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.814950 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-socket\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815284 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.815023 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istiod-ca-cert\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.815319 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-socket\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815493 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.815454 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-envoy\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.815523 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-certs\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815615 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.815557 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-credential-socket\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.815641 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-podinfo\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.815732 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.815667 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-token\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.844768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.816634 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istiod-ca-cert\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.844768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.817147 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-credential-socket\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.844768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.817387 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-certs\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.844768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.841337 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-envoy\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.844768 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.844554 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-podinfo\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.861617 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.861578 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:56.882145 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.882076 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbsx\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-kube-api-access-lzbsx\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.885807 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.885782 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-token\") pod \"e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:56.932434 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:56.932339 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:57.359795 ip-10-0-137-225 kubenswrapper[4441]: W1102 21:59:57.359766 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aac1cb0_9792_4d93_b66e_259a1d6a6500.slice/crio-0ae2c4ed015d90fdcf6d929c9a4f24b52f1849cd0ef95ce2fb898a40516b4761 WatchSource:0}: Error finding container 0ae2c4ed015d90fdcf6d929c9a4f24b52f1849cd0ef95ce2fb898a40516b4761: Status 404 returned error can't find the container with id 0ae2c4ed015d90fdcf6d929c9a4f24b52f1849cd0ef95ce2fb898a40516b4761 Nov 02 21:59:57.360170 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:57.360148 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7"] Nov 02 21:59:57.362348 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:57.362314 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:59:57.362447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:57.362378 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:59:57.362447 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:57.362409 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 21:59:57.880362 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:57.879995 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:59:57.930431 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:57.930403 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:57.939937 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:57.939908 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:59:58.022935 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.022900 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:59:58.031221 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.031194 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:59:58.040579 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040542 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-certs\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.040689 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040629 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-socket\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.040726 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040686 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-envoy\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.040817 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040769 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-token\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.040864 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040826 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-data\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.040899 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040871 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-certs\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.040936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040900 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e0b4b6-6923-42f2-8595-6855d32a0053-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0b4b6-6923-42f2-8595-6855d32a0053-istiod-ca-cert\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.040936 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040906 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-certs" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.041038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040952 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-socket" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.041038 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.040955 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-credential-socket\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.041104 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041056 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-kube-api-access-7spnr\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-kube-api-access-7spnr\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.041139 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041121 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-credential-socket\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.041222 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041205 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-podinfo\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.041320 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041304 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-podinfo\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.041385 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041344 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d28baed-dc19-4f33-803a-8a19e68d50c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d28baed-dc19-4f33-803a-8a19e68d50c0-istiod-ca-cert\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.041446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041404 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-socket\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.041446 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041437 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-envoy\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.041569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041466 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-kube-api-access-lchjz\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-kube-api-access-lchjz\") pod \"d4e0b4b6-6923-42f2-8595-6855d32a0053\" (UID: \"d4e0b4b6-6923-42f2-8595-6855d32a0053\") " Nov 02 21:59:58.041569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041504 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-data\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.041569 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041530 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-token\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-token\") pod \"5d28baed-dc19-4f33-803a-8a19e68d50c0\" (UID: \"5d28baed-dc19-4f33-803a-8a19e68d50c0\") " Nov 02 21:59:58.042397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041206 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-credential-socket" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.042397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.042164 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.042397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.042188 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.042397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041429 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-data" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.042397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.041508 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-credential-socket" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.042397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.042091 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-certs" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.042397 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.042140 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e0b4b6-6923-42f2-8595-6855d32a0053-istiod-ca-cert" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:58.042867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.042419 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-data" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.042867 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.042831 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-socket" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.043203 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.043175 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d28baed-dc19-4f33-803a-8a19e68d50c0-istiod-ca-cert" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:58.044140 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.044062 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-envoy" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.044255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.044171 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-kube-api-access-7spnr" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "kube-api-access-7spnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.046058 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.045716 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-envoy" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.047448 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.047422 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-kube-api-access-lchjz" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "kube-api-access-lchjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.047580 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.047560 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-token" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.047650 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.047572 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-podinfo" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:59:58.048457 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.048437 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-podinfo" pod "d4e0b4b6-6923-42f2-8595-6855d32a0053" (UID: "d4e0b4b6-6923-42f2-8595-6855d32a0053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:59:58.049303 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.049268 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-token" pod "5d28baed-dc19-4f33-803a-8a19e68d50c0" (UID: "5d28baed-dc19-4f33-803a-8a19e68d50c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.143346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143260 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-certs\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.143346 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143318 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-podinfo\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143371 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-envoy\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143404 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-credential-socket\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.143591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143463 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-kube-api-access-pm4t6\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-kube-api-access-pm4t6\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.143591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143495 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-credential-socket\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143540 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-token\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-token\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143566 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-certs" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143605 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-socket\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143635 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-socket\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143707 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-kube-api-access-7g9qt\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-kube-api-access-7g9qt\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143740 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-data\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143810 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-token\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-token\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143867 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istiod-ca-cert\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143915 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istiod-ca-cert\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.143953 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.143950 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-certs\") pod \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\" (UID: \"fdf0e9e9-98cd-4e04-a71a-e028bdb73713\") " Nov 02 21:59:58.144476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144002 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-data\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.144476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144074 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-podinfo\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.144476 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144156 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-envoy\") pod \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\" (UID: \"d4e7d807-33c6-49a7-801a-1e1be51c7eab\") " Nov 02 21:59:58.144749 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144727 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.144813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144755 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.144813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144770 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d28baed-dc19-4f33-803a-8a19e68d50c0-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.144813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144771 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-certs" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.144813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144783 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.144813 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144799 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144813 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lchjz\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-kube-api-access-lchjz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144828 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144842 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144856 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144869 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144883 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144898 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144912 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0b4b6-6923-42f2-8595-6855d32a0053-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144926 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d28baed-dc19-4f33-803a-8a19e68d50c0-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144941 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7spnr\" (UniqueName: \"kubernetes.io/projected/5d28baed-dc19-4f33-803a-8a19e68d50c0-kube-api-access-7spnr\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144957 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0b4b6-6923-42f2-8595-6855d32a0053-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.144996 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d28baed-dc19-4f33-803a-8a19e68d50c0-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.145130 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.145008 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-credential-socket" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.145847 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.145335 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istiod-ca-cert" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:58.146005 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.145981 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-credential-socket" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.146399 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.146369 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istiod-ca-cert" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:58.146570 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.146545 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerID="889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48" exitCode=0 Nov 02 21:59:58.146676 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.146659 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" event={"ID":"d4e7d807-33c6-49a7-801a-1e1be51c7eab","Type":"ContainerDied","Data":"889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48"} Nov 02 21:59:58.146747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.146703 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" event={"ID":"d4e7d807-33c6-49a7-801a-1e1be51c7eab","Type":"ContainerDied","Data":"3d59ebd75cfac3c7f56960049494a6c0a63f47d771d03d93e496d14147af8474"} Nov 02 21:59:58.146747 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.146729 4441 scope.go:122] "RemoveContainer" containerID="889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48" Nov 02 21:59:58.146869 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.146853 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj" Nov 02 21:59:58.147526 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.147285 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-socket" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.147646 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.147541 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-token" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.147921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.147726 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-data" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.147921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.147740 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-envoy" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.147921 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.147755 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-socket" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.148183 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.148003 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-data" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.148489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.148441 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-podinfo" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:59:58.148489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.148460 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-envoy" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:58.148489 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.148472 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-kube-api-access-pm4t6" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "kube-api-access-pm4t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.149531 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.149504 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-kube-api-access-7g9qt" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "kube-api-access-7g9qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.150120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.149957 4441 generic.go:356] "Generic (PLEG): container finished" podID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerID="138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9" exitCode=0 Nov 02 21:59:58.150120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.150027 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" Nov 02 21:59:58.150120 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.150082 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" event={"ID":"5d28baed-dc19-4f33-803a-8a19e68d50c0","Type":"ContainerDied","Data":"138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9"} Nov 02 21:59:58.150339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.150123 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g" event={"ID":"5d28baed-dc19-4f33-803a-8a19e68d50c0","Type":"ContainerDied","Data":"5aee4b0149f02b0b4041f85498dec5469ecff25cc86e3e1d02044db0f541471d"} Nov 02 21:59:58.150339 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.150174 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-token" pod "fdf0e9e9-98cd-4e04-a71a-e028bdb73713" (UID: "fdf0e9e9-98cd-4e04-a71a-e028bdb73713"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.151255 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.151213 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-podinfo" pod "d4e7d807-33c6-49a7-801a-1e1be51c7eab" (UID: "d4e7d807-33c6-49a7-801a-1e1be51c7eab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:59:58.151830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.151718 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerID="4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee" exitCode=0 Nov 02 21:59:58.151830 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.151766 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" Nov 02 21:59:58.152029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.151865 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" event={"ID":"d4e0b4b6-6923-42f2-8595-6855d32a0053","Type":"ContainerDied","Data":"4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee"} Nov 02 21:59:58.152029 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.151905 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr" event={"ID":"d4e0b4b6-6923-42f2-8595-6855d32a0053","Type":"ContainerDied","Data":"bbe2e8f50385c5f9b058d21de969919faff5845c7d4a66ec806c44f60c49b70c"} Nov 02 21:59:58.153799 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.153767 4441 generic.go:356] "Generic (PLEG): container finished" podID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerID="a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495" exitCode=0 Nov 02 21:59:58.153902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.153813 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" Nov 02 21:59:58.153902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.153847 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" event={"ID":"fdf0e9e9-98cd-4e04-a71a-e028bdb73713","Type":"ContainerDied","Data":"a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495"} Nov 02 21:59:58.153902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.153886 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2" event={"ID":"fdf0e9e9-98cd-4e04-a71a-e028bdb73713","Type":"ContainerDied","Data":"d66fb3b708e0ccf74ca6f32848c1bc435bd234331c528dcbbbe1c258dc2f1b12"} Nov 02 21:59:58.156444 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.156418 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" event={"ID":"2aac1cb0-9792-4d93-b66e-259a1d6a6500","Type":"ContainerStarted","Data":"f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc"} Nov 02 21:59:58.156560 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.156465 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" event={"ID":"2aac1cb0-9792-4d93-b66e-259a1d6a6500","Type":"ContainerStarted","Data":"0ae2c4ed015d90fdcf6d929c9a4f24b52f1849cd0ef95ce2fb898a40516b4761"} Nov 02 21:59:58.163117 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.163093 4441 scope.go:122] "RemoveContainer" containerID="889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48" Nov 02 21:59:58.163465 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:59:58.163436 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48\": container with ID starting with 889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48 not found: ID does not exist" containerID="889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48" Nov 02 21:59:58.163555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.163471 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48"} err="failed to get container status \"889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48\": rpc error: code = NotFound desc = could not find container \"889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48\": container with ID starting with 889beabf3d2be0bc438a9aea562b09f72772b669118d09a6ddb5a2a4af3efc48 not found: ID does not exist" Nov 02 21:59:58.163555 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.163495 4441 scope.go:122] "RemoveContainer" containerID="138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9" Nov 02 21:59:58.173616 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.173597 4441 scope.go:122] "RemoveContainer" containerID="138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9" Nov 02 21:59:58.173880 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:59:58.173862 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9\": container with ID starting with 138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9 not found: ID does not exist" containerID="138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9" Nov 02 21:59:58.173947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.173886 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9"} err="failed to get container status \"138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9\": rpc error: code = NotFound desc = could not find container \"138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9\": container with ID starting with 138524d362ea3ddd0b773b4b3bdb464df918304385756a494ace2afadbf1c3f9 not found: ID does not exist" Nov 02 21:59:58.173947 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.173901 4441 scope.go:122] "RemoveContainer" containerID="4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee" Nov 02 21:59:58.183268 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.183253 4441 scope.go:122] "RemoveContainer" containerID="4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee" Nov 02 21:59:58.183505 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:59:58.183488 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee\": container with ID starting with 4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee not found: ID does not exist" containerID="4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee" Nov 02 21:59:58.183552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.183510 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee"} err="failed to get container status \"4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee\": rpc error: code = NotFound desc = could not find container \"4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee\": container with ID starting with 4fd416f471cdb9f385a455d8bab5339bc7e5f516fae7107ced88221d065062ee not found: ID does not exist" Nov 02 21:59:58.183552 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.183527 4441 scope.go:122] "RemoveContainer" containerID="a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495" Nov 02 21:59:58.192225 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.192197 4441 scope.go:122] "RemoveContainer" containerID="a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495" Nov 02 21:59:58.192477 ip-10-0-137-225 kubenswrapper[4441]: E1102 21:59:58.192453 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495\": container with ID starting with a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495 not found: ID does not exist" containerID="a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495" Nov 02 21:59:58.192591 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.192485 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495"} err="failed to get container status \"a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495\": rpc error: code = NotFound desc = could not find container \"a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495\": container with ID starting with a9824c749279afa450496e1f40fb76d8307eacf64ba060029b3fb883c6061495 not found: ID does not exist" Nov 02 21:59:58.247100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247060 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247100 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247097 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247112 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247126 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247139 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247152 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm4t6\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-kube-api-access-pm4t6\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247168 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247181 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247194 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247207 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247221 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9qt\" (UniqueName: \"kubernetes.io/projected/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-kube-api-access-7g9qt\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247235 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247249 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247264 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247276 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247287 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247290 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf0e9e9-98cd-4e04-a71a-e028bdb73713-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.247742 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.247304 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e7d807-33c6-49a7-801a-1e1be51c7eab-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.445263 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.445186 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2"] Nov 02 21:59:58.450323 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.450293 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rp6l2"] Nov 02 21:59:58.554033 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.554004 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g"] Nov 02 21:59:58.567039 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.560557 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kch8g"] Nov 02 21:59:58.583865 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.583835 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" path="/var/lib/kubelet/pods/5d28baed-dc19-4f33-803a-8a19e68d50c0/volumes" Nov 02 21:59:58.584345 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.584330 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" path="/var/lib/kubelet/pods/fdf0e9e9-98cd-4e04-a71a-e028bdb73713/volumes" Nov 02 21:59:58.664938 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.664892 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" podStartSLOduration=2.664879267 podStartE2EDuration="2.664879267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:58.663671397 +0000 UTC m=+2828.219299734" watchObservedRunningTime="2026-11-02 21:59:58.664879267 +0000 UTC m=+2828.220507603" Nov 02 21:59:58.770902 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.770828 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr"] Nov 02 21:59:58.774347 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.774308 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s5kwr"] Nov 02 21:59:58.926429 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.926389 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj"] Nov 02 21:59:58.932085 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.932058 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-662qj"] Nov 02 21:59:58.933124 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.933103 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:58.938496 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:58.938474 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:59.167992 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:59.167941 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 21:59:59.168796 ip-10-0-137-225 kubenswrapper[4441]: I1102 21:59:59.168779 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 22:00:00.156697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.156657 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw"] Nov 02 22:00:00.157524 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157499 4441 cpu_manager.go:414] "containerMap: removing container" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" Nov 02 22:00:00.157719 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157694 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" Nov 02 22:00:00.157719 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157725 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157735 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157746 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157753 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157832 4441 cpu_manager.go:414] "containerMap: removing container" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.157839 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.158033 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d28baed-dc19-4f33-803a-8a19e68d50c0" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.158046 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf0e9e9-98cd-4e04-a71a-e028bdb73713" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.158056 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" containerName="istio-proxy" Nov 02 22:00:00.158238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.158070 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" containerName="istio-proxy" Nov 02 22:00:00.163886 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.163865 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv"] Nov 02 22:00:00.164018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.163999 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.167861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.167835 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7"] Nov 02 22:00:00.167959 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.167921 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.171911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.171886 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.203337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.203311 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6"] Nov 02 22:00:00.207355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.207328 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd"] Nov 02 22:00:00.207741 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.207721 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.211743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.211722 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.211915 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.211865 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg"] Nov 02 22:00:00.216694 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.216670 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.217738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.217722 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9"] Nov 02 22:00:00.222432 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.222413 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.222857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.222739 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl"] Nov 02 22:00:00.226889 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.226869 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l"] Nov 02 22:00:00.229115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.228185 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.234452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.234429 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.241933 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.241912 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb"] Nov 02 22:00:00.246200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.246182 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.251398 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.251374 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:00:00.251398 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.251382 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:00:00.251650 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.251548 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:00:00.257485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.252543 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw"] Nov 02 22:00:00.258856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.258834 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv"] Nov 02 22:00:00.263222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.263201 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6"] Nov 02 22:00:00.274698 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.274673 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7"] Nov 02 22:00:00.274987 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.274952 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd"] Nov 02 22:00:00.282161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282131 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27x9c\" (UniqueName: \"kubernetes.io/projected/a9b57b7e-4b68-434c-a360-5ce0f620a833-kube-api-access-27x9c\") pod \"maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.282286 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282208 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq96k\" (UniqueName: \"kubernetes.io/projected/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-kube-api-access-dq96k\") pod \"maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.282286 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282263 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fe3e8-170b-4f22-922d-9c8633eb9379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.282418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282336 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmnkq\" (UniqueName: \"kubernetes.io/projected/7e8fe3e8-170b-4f22-922d-9c8633eb9379-kube-api-access-zmnkq\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.282484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282363 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr7hx\" (UniqueName: \"kubernetes.io/projected/659c34e8-7543-443e-9600-c049cf185952-kube-api-access-hr7hx\") pod \"maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.282535 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282502 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fw6\" (UniqueName: \"kubernetes.io/projected/975396b8-867d-479b-8a00-7c6e235414f6-kube-api-access-z2fw6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.282802 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282785 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.283067 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.282811 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975396b8-867d-479b-8a00-7c6e235414f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.283341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.283323 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c537f1-c0de-45d3-8156-f013aa218edf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.283550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.283534 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5c852-2c53-4e74-babf-0f702210717f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.283750 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.283736 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m278t\" (UniqueName: \"kubernetes.io/projected/53c537f1-c0de-45d3-8156-f013aa218edf-kube-api-access-m278t\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.283960 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.283780 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wnd\" (UniqueName: \"kubernetes.io/projected/05a5c852-2c53-4e74-babf-0f702210717f-kube-api-access-z6wnd\") pod \"maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.284058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.284001 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cc9c91-4155-413b-989f-4bc18e1de331-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-hgdrg\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.284248 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.284231 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq4pb\" (UniqueName: \"kubernetes.io/projected/15c454cc-a1c8-4b15-98b3-f6a61c708956-kube-api-access-pq4pb\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.284448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.284431 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c34e8-7543-443e-9600-c049cf185952-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.284547 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.284487 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae81538-0e22-463d-9394-f45701065e72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.284662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.284593 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c454cc-a1c8-4b15-98b3-f6a61c708956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.284728 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.284708 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjw4j\" (UniqueName: \"kubernetes.io/projected/7ae81538-0e22-463d-9394-f45701065e72-kube-api-access-kjw4j\") pod \"maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.284838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.284823 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b57b7e-4b68-434c-a360-5ce0f620a833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.298701 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.295823 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm66k\" (UniqueName: \"kubernetes.io/projected/e7cc9c91-4155-413b-989f-4bc18e1de331-kube-api-access-cm66k\") pod \"maas-api-key-cleanup-29894280-hgdrg\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.321721 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.321687 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl"] Nov 02 22:00:00.322001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.321984 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb"] Nov 02 22:00:00.344505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.343438 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9"] Nov 02 22:00:00.354641 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.354611 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg"] Nov 02 22:00:00.360336 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.360311 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l"] Nov 02 22:00:00.396789 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.396758 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae81538-0e22-463d-9394-f45701065e72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.396928 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.396798 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c454cc-a1c8-4b15-98b3-f6a61c708956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.396928 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.396830 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjw4j\" (UniqueName: \"kubernetes.io/projected/7ae81538-0e22-463d-9394-f45701065e72-kube-api-access-kjw4j\") pod \"maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.397076 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.396929 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b57b7e-4b68-434c-a360-5ce0f620a833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.397076 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397002 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm66k\" (UniqueName: \"kubernetes.io/projected/e7cc9c91-4155-413b-989f-4bc18e1de331-kube-api-access-cm66k\") pod \"maas-api-key-cleanup-29894280-hgdrg\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.397076 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397033 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27x9c\" (UniqueName: \"kubernetes.io/projected/a9b57b7e-4b68-434c-a360-5ce0f620a833-kube-api-access-27x9c\") pod \"maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.397239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397083 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq96k\" (UniqueName: \"kubernetes.io/projected/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-kube-api-access-dq96k\") pod \"maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.397239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397100 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fe3e8-170b-4f22-922d-9c8633eb9379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.397239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397147 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnkq\" (UniqueName: \"kubernetes.io/projected/7e8fe3e8-170b-4f22-922d-9c8633eb9379-kube-api-access-zmnkq\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.397239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397182 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr7hx\" (UniqueName: \"kubernetes.io/projected/659c34e8-7543-443e-9600-c049cf185952-kube-api-access-hr7hx\") pod \"maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.397239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397221 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fw6\" (UniqueName: \"kubernetes.io/projected/975396b8-867d-479b-8a00-7c6e235414f6-kube-api-access-z2fw6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.397531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397277 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.397531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397305 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975396b8-867d-479b-8a00-7c6e235414f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.397531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397344 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c537f1-c0de-45d3-8156-f013aa218edf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.397531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397373 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5c852-2c53-4e74-babf-0f702210717f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.397531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397401 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m278t\" (UniqueName: \"kubernetes.io/projected/53c537f1-c0de-45d3-8156-f013aa218edf-kube-api-access-m278t\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.397531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397458 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wnd\" (UniqueName: \"kubernetes.io/projected/05a5c852-2c53-4e74-babf-0f702210717f-kube-api-access-z6wnd\") pod \"maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.397531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397495 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cc9c91-4155-413b-989f-4bc18e1de331-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-hgdrg\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.397911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397547 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq4pb\" (UniqueName: \"kubernetes.io/projected/15c454cc-a1c8-4b15-98b3-f6a61c708956-kube-api-access-pq4pb\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.397911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397586 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c34e8-7543-443e-9600-c049cf185952-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.397911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397670 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae81538-0e22-463d-9394-f45701065e72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.397911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397688 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c454cc-a1c8-4b15-98b3-f6a61c708956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.397911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.397767 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b57b7e-4b68-434c-a360-5ce0f620a833-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.398293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.398042 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fe3e8-170b-4f22-922d-9c8633eb9379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.398293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.398153 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975396b8-867d-479b-8a00-7c6e235414f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.398293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.398205 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.398473 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.398451 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cc9c91-4155-413b-989f-4bc18e1de331-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-hgdrg\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.398546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.398456 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c537f1-c0de-45d3-8156-f013aa218edf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.398664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.398641 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c34e8-7543-443e-9600-c049cf185952-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.398731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.398646 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5c852-2c53-4e74-babf-0f702210717f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.453027 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.452930 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjw4j\" (UniqueName: \"kubernetes.io/projected/7ae81538-0e22-463d-9394-f45701065e72-kube-api-access-kjw4j\") pod \"maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.453027 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.453015 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m278t\" (UniqueName: \"kubernetes.io/projected/53c537f1-c0de-45d3-8156-f013aa218edf-kube-api-access-m278t\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.453245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.453029 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmnkq\" (UniqueName: \"kubernetes.io/projected/7e8fe3e8-170b-4f22-922d-9c8633eb9379-kube-api-access-zmnkq\") pod \"maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.453245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.453012 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fw6\" (UniqueName: \"kubernetes.io/projected/975396b8-867d-479b-8a00-7c6e235414f6-kube-api-access-z2fw6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.453245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.453067 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq96k\" (UniqueName: \"kubernetes.io/projected/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-kube-api-access-dq96k\") pod \"maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.453245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.453084 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq4pb\" (UniqueName: \"kubernetes.io/projected/15c454cc-a1c8-4b15-98b3-f6a61c708956-kube-api-access-pq4pb\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.453245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.453159 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27x9c\" (UniqueName: \"kubernetes.io/projected/a9b57b7e-4b68-434c-a360-5ce0f620a833-kube-api-access-27x9c\") pod \"maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.453728 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.453712 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr7hx\" (UniqueName: \"kubernetes.io/projected/659c34e8-7543-443e-9600-c049cf185952-kube-api-access-hr7hx\") pod \"maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.455125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.455108 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm66k\" (UniqueName: \"kubernetes.io/projected/e7cc9c91-4155-413b-989f-4bc18e1de331-kube-api-access-cm66k\") pod \"maas-api-key-cleanup-29894280-hgdrg\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.455572 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.455550 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wnd\" (UniqueName: \"kubernetes.io/projected/05a5c852-2c53-4e74-babf-0f702210717f-kube-api-access-z6wnd\") pod \"maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.480297 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.480274 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:00.487934 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.487916 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:00.494072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.494047 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:00.536287 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.536247 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:00.548217 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.548184 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:00.565983 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.565727 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:00.603752 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.603714 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e0b4b6-6923-42f2-8595-6855d32a0053" path="/var/lib/kubelet/pods/d4e0b4b6-6923-42f2-8595-6855d32a0053/volumes" Nov 02 22:00:00.604462 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.604439 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e7d807-33c6-49a7-801a-1e1be51c7eab" path="/var/lib/kubelet/pods/d4e7d807-33c6-49a7-801a-1e1be51c7eab/volumes" Nov 02 22:00:00.605385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.605173 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:00.639460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.636213 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:00.652998 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.652761 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:00.679578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.679063 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:00.870836 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:00.870807 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw"] Nov 02 22:00:01.103102 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.103078 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7"] Nov 02 22:00:01.105124 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:01.105092 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c537f1_c0de_45d3_8156_f013aa218edf.slice/crio-f0bc585d9ca7d22cc9c51491be6baad70e86ee167ece8fa352df2bdff6d00d55 WatchSource:0}: Error finding container f0bc585d9ca7d22cc9c51491be6baad70e86ee167ece8fa352df2bdff6d00d55: Status 404 returned error can't find the container with id f0bc585d9ca7d22cc9c51491be6baad70e86ee167ece8fa352df2bdff6d00d55 Nov 02 22:00:01.115496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.115461 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv"] Nov 02 22:00:01.149598 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.149573 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb"] Nov 02 22:00:01.163848 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.163582 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l"] Nov 02 22:00:01.168191 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.168165 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl"] Nov 02 22:00:01.178248 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:01.178220 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b57b7e_4b68_434c_a360_5ce0f620a833.slice/crio-ec40f7f3631298630ceb3a5a84efc09dacd717122f618b22977c7b944c9b254f WatchSource:0}: Error finding container ec40f7f3631298630ceb3a5a84efc09dacd717122f618b22977c7b944c9b254f: Status 404 returned error can't find the container with id ec40f7f3631298630ceb3a5a84efc09dacd717122f618b22977c7b944c9b254f Nov 02 22:00:01.198806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.198774 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" event={"ID":"7ae81538-0e22-463d-9394-f45701065e72","Type":"ContainerStarted","Data":"e3736e0cf88c13cf21fd1ce20c8d2039164b7c919a0edd06925203ac287ccc73"} Nov 02 22:00:01.201463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.201301 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6"] Nov 02 22:00:01.205659 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.205633 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd"] Nov 02 22:00:01.206038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.205932 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerStarted","Data":"f0bc585d9ca7d22cc9c51491be6baad70e86ee167ece8fa352df2bdff6d00d55"} Nov 02 22:00:01.209444 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:01.209415 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975396b8_867d_479b_8a00_7c6e235414f6.slice/crio-d50fc2cfca8acf9f5c977587ea880b4085901f8138ec5fa5eaa13f191213c82e WatchSource:0}: Error finding container d50fc2cfca8acf9f5c977587ea880b4085901f8138ec5fa5eaa13f191213c82e: Status 404 returned error can't find the container with id d50fc2cfca8acf9f5c977587ea880b4085901f8138ec5fa5eaa13f191213c82e Nov 02 22:00:01.222638 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.222612 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg"] Nov 02 22:00:01.222748 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:01.222637 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a5c852_2c53_4e74_babf_0f702210717f.slice/crio-94ffd071296d980827b5d7225bb0d64dd2e965510cc2a813a21f2a648e0afd75 WatchSource:0}: Error finding container 94ffd071296d980827b5d7225bb0d64dd2e965510cc2a813a21f2a648e0afd75: Status 404 returned error can't find the container with id 94ffd071296d980827b5d7225bb0d64dd2e965510cc2a813a21f2a648e0afd75 Nov 02 22:00:01.232104 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:01.232081 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cc9c91_4155_413b_989f_4bc18e1de331.slice/crio-b6e7a91cd13ea3efd26c68ee47a7c217073a3fcac1bcb01ffe52e0a3fb2fd7bf WatchSource:0}: Error finding container b6e7a91cd13ea3efd26c68ee47a7c217073a3fcac1bcb01ffe52e0a3fb2fd7bf: Status 404 returned error can't find the container with id b6e7a91cd13ea3efd26c68ee47a7c217073a3fcac1bcb01ffe52e0a3fb2fd7bf Nov 02 22:00:01.242795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.242773 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9"] Nov 02 22:00:01.250230 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:01.250204 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c454cc_a1c8_4b15_98b3_f6a61c708956.slice/crio-92ba0597e50dc7c1ab41d87dd79b1483133fdef371f621d330da2efe67e2e764 WatchSource:0}: Error finding container 92ba0597e50dc7c1ab41d87dd79b1483133fdef371f621d330da2efe67e2e764: Status 404 returned error can't find the container with id 92ba0597e50dc7c1ab41d87dd79b1483133fdef371f621d330da2efe67e2e764 Nov 02 22:00:01.396625 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.396592 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:01.561881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.561850 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:01.814699 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:01.814675 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:02.215170 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.215131 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerStarted","Data":"e084862cfafec26fc8908a48f5a8ab8884759ff2ec486861f0f1c2732d73f270"} Nov 02 22:00:02.215617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.215197 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerStarted","Data":"ec40f7f3631298630ceb3a5a84efc09dacd717122f618b22977c7b944c9b254f"} Nov 02 22:00:02.217111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.217071 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerStarted","Data":"74ae9aba8eb667f96ed3db3aa9a1d9b348af5f79d32bf7c834f26e40debae40e"} Nov 02 22:00:02.217231 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.217131 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerStarted","Data":"d50fc2cfca8acf9f5c977587ea880b4085901f8138ec5fa5eaa13f191213c82e"} Nov 02 22:00:02.218765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.218736 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerStarted","Data":"4a1ae6cda7aa2b2c7eb063888247153efd735d9daa19234541e4d30515e132f5"} Nov 02 22:00:02.218902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.218781 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerStarted","Data":"a98161d0935b8bfef32e80fb76f45f4e8d3230cea57167b692abff1beaef48f6"} Nov 02 22:00:02.220476 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.220442 4441 generic.go:356] "Generic (PLEG): container finished" podID="7ae81538-0e22-463d-9394-f45701065e72" containerID="4648ccd118ea93d45a0ac4076d78848f1d83da6e9b1ce5fcbace6d2d95746467" exitCode=60 Nov 02 22:00:02.220585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.220528 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" event={"ID":"7ae81538-0e22-463d-9394-f45701065e72","Type":"ContainerDied","Data":"4648ccd118ea93d45a0ac4076d78848f1d83da6e9b1ce5fcbace6d2d95746467"} Nov 02 22:00:02.220699 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.220685 4441 scope.go:122] "RemoveContainer" containerID="4648ccd118ea93d45a0ac4076d78848f1d83da6e9b1ce5fcbace6d2d95746467" Nov 02 22:00:02.222233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.222121 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerStarted","Data":"5cfcb8c575e1330f31a99a3457d56631cda18a38647255fe236f05ab92f52d34"} Nov 02 22:00:02.222233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.222170 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerStarted","Data":"b6e7a91cd13ea3efd26c68ee47a7c217073a3fcac1bcb01ffe52e0a3fb2fd7bf"} Nov 02 22:00:02.223665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.223641 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerStarted","Data":"76f9428742afc2b75be7352828c8465fe774b087f70c986af4278d4472a9fbba"} Nov 02 22:00:02.225546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.225522 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerStarted","Data":"236bebe2f88ca13336b3c2a88975daccb204b49d7db961a529d1e46923929136"} Nov 02 22:00:02.225640 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.225566 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerStarted","Data":"92ba0597e50dc7c1ab41d87dd79b1483133fdef371f621d330da2efe67e2e764"} Nov 02 22:00:02.227106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.227079 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerStarted","Data":"8660baa7d27dc9594041c0dfa37f62be5000a68cf8bd42b659f552bc63ed355e"} Nov 02 22:00:02.227213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.227129 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerStarted","Data":"94ffd071296d980827b5d7225bb0d64dd2e965510cc2a813a21f2a648e0afd75"} Nov 02 22:00:02.228867 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.228845 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerStarted","Data":"22559ee183442a740c20fea0ee73444f28bb7c7b0d5baee7c2e3c52203de7516"} Nov 02 22:00:02.228951 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.228884 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerStarted","Data":"5c5143507d8442b1e59daec9dbe1ecec13e395c30863f3f8a695ded917fa1855"} Nov 02 22:00:02.230438 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.230418 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerStarted","Data":"68ab98ac4367b14e8d8c296f7002826bdcf85635cfc786fc79b624e9fac820cb"} Nov 02 22:00:02.230542 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.230450 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerStarted","Data":"d7b74e07aa2e4e6ab19657775a86a01f3eeb505084107a330cf22a7379bc5dfd"} Nov 02 22:00:02.330259 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.330208 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" podStartSLOduration=2.330193992 podStartE2EDuration="2.330193992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.324801139 +0000 UTC m=+2831.880429478" watchObservedRunningTime="2026-11-02 22:00:02.330193992 +0000 UTC m=+2831.885822329" Nov 02 22:00:02.437073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.437020 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" podStartSLOduration=2.437006285 podStartE2EDuration="2.437006285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.433203422 +0000 UTC m=+2831.988831760" watchObservedRunningTime="2026-11-02 22:00:02.437006285 +0000 UTC m=+2831.992634619" Nov 02 22:00:02.574211 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.574170 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:02.650279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.650232 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" podStartSLOduration=2.650218218 podStartE2EDuration="2.650218218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.648653389 +0000 UTC m=+2832.204281725" watchObservedRunningTime="2026-11-02 22:00:02.650218218 +0000 UTC m=+2832.205846552" Nov 02 22:00:02.760397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.760347 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" podStartSLOduration=2.760333787 podStartE2EDuration="2.760333787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.756831215 +0000 UTC m=+2832.312459553" watchObservedRunningTime="2026-11-02 22:00:02.760333787 +0000 UTC m=+2832.315962126" Nov 02 22:00:02.867263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.867212 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" podStartSLOduration=2.8671975830000003 podStartE2EDuration="2.867197583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.866115313 +0000 UTC m=+2832.421743651" watchObservedRunningTime="2026-11-02 22:00:02.867197583 +0000 UTC m=+2832.422825920" Nov 02 22:00:02.976432 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:02.976319 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" podStartSLOduration=2.976304921 podStartE2EDuration="2.976304921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.972586445 +0000 UTC m=+2832.528214782" watchObservedRunningTime="2026-11-02 22:00:02.976304921 +0000 UTC m=+2832.531933321" Nov 02 22:00:03.081071 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.081013 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" podStartSLOduration=3.080999308 podStartE2EDuration="3.080999308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.079448103 +0000 UTC m=+2832.635076453" watchObservedRunningTime="2026-11-02 22:00:03.080999308 +0000 UTC m=+2832.636627645" Nov 02 22:00:03.193430 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.193376 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" podStartSLOduration=3.193362026 podStartE2EDuration="3.193362026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.189120043 +0000 UTC m=+2832.744748391" watchObservedRunningTime="2026-11-02 22:00:03.193362026 +0000 UTC m=+2832.748990363" Nov 02 22:00:03.242853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.242766 4441 generic.go:356] "Generic (PLEG): container finished" podID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerID="68ab98ac4367b14e8d8c296f7002826bdcf85635cfc786fc79b624e9fac820cb" exitCode=60 Nov 02 22:00:03.243329 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.242853 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerDied","Data":"68ab98ac4367b14e8d8c296f7002826bdcf85635cfc786fc79b624e9fac820cb"} Nov 02 22:00:03.243329 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.243136 4441 scope.go:122] "RemoveContainer" containerID="68ab98ac4367b14e8d8c296f7002826bdcf85635cfc786fc79b624e9fac820cb" Nov 02 22:00:03.244894 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.244866 4441 generic.go:356] "Generic (PLEG): container finished" podID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerID="e084862cfafec26fc8908a48f5a8ab8884759ff2ec486861f0f1c2732d73f270" exitCode=60 Nov 02 22:00:03.245037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.244951 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerDied","Data":"e084862cfafec26fc8908a48f5a8ab8884759ff2ec486861f0f1c2732d73f270"} Nov 02 22:00:03.245215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.245196 4441 scope.go:122] "RemoveContainer" containerID="e084862cfafec26fc8908a48f5a8ab8884759ff2ec486861f0f1c2732d73f270" Nov 02 22:00:03.246961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.246937 4441 generic.go:356] "Generic (PLEG): container finished" podID="975396b8-867d-479b-8a00-7c6e235414f6" containerID="74ae9aba8eb667f96ed3db3aa9a1d9b348af5f79d32bf7c834f26e40debae40e" exitCode=60 Nov 02 22:00:03.246961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.246980 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerDied","Data":"74ae9aba8eb667f96ed3db3aa9a1d9b348af5f79d32bf7c834f26e40debae40e"} Nov 02 22:00:03.247296 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.247276 4441 scope.go:122] "RemoveContainer" containerID="74ae9aba8eb667f96ed3db3aa9a1d9b348af5f79d32bf7c834f26e40debae40e" Nov 02 22:00:03.249784 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.249762 4441 generic.go:356] "Generic (PLEG): container finished" podID="659c34e8-7543-443e-9600-c049cf185952" containerID="4a1ae6cda7aa2b2c7eb063888247153efd735d9daa19234541e4d30515e132f5" exitCode=60 Nov 02 22:00:03.249920 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.249897 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerDied","Data":"4a1ae6cda7aa2b2c7eb063888247153efd735d9daa19234541e4d30515e132f5"} Nov 02 22:00:03.250108 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.250089 4441 scope.go:122] "RemoveContainer" containerID="4a1ae6cda7aa2b2c7eb063888247153efd735d9daa19234541e4d30515e132f5" Nov 02 22:00:03.251906 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.251886 4441 generic.go:356] "Generic (PLEG): container finished" podID="7ae81538-0e22-463d-9394-f45701065e72" containerID="2a06249f7d99a96720f1049fba0842663fb8fc09d745a8f31e8099d25587595f" exitCode=60 Nov 02 22:00:03.252026 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.251910 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" event={"ID":"7ae81538-0e22-463d-9394-f45701065e72","Type":"ContainerDied","Data":"2a06249f7d99a96720f1049fba0842663fb8fc09d745a8f31e8099d25587595f"} Nov 02 22:00:03.252026 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.251962 4441 scope.go:122] "RemoveContainer" containerID="4648ccd118ea93d45a0ac4076d78848f1d83da6e9b1ce5fcbace6d2d95746467" Nov 02 22:00:03.252213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.252196 4441 scope.go:122] "RemoveContainer" containerID="2a06249f7d99a96720f1049fba0842663fb8fc09d745a8f31e8099d25587595f" Nov 02 22:00:03.252437 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:03.252414 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw_odh-ai-gateway-infra(7ae81538-0e22-463d-9394-f45701065e72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" podUID="7ae81538-0e22-463d-9394-f45701065e72" Nov 02 22:00:03.253621 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.253595 4441 generic.go:356] "Generic (PLEG): container finished" podID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerID="5cfcb8c575e1330f31a99a3457d56631cda18a38647255fe236f05ab92f52d34" exitCode=60 Nov 02 22:00:03.253728 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.253675 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerDied","Data":"5cfcb8c575e1330f31a99a3457d56631cda18a38647255fe236f05ab92f52d34"} Nov 02 22:00:03.253850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.253833 4441 scope.go:122] "RemoveContainer" containerID="5cfcb8c575e1330f31a99a3457d56631cda18a38647255fe236f05ab92f52d34" Nov 02 22:00:03.255785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.255659 4441 generic.go:356] "Generic (PLEG): container finished" podID="53c537f1-c0de-45d3-8156-f013aa218edf" containerID="76f9428742afc2b75be7352828c8465fe774b087f70c986af4278d4472a9fbba" exitCode=60 Nov 02 22:00:03.255785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.255725 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerDied","Data":"76f9428742afc2b75be7352828c8465fe774b087f70c986af4278d4472a9fbba"} Nov 02 22:00:03.256189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.256038 4441 scope.go:122] "RemoveContainer" containerID="76f9428742afc2b75be7352828c8465fe774b087f70c986af4278d4472a9fbba" Nov 02 22:00:03.257458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.257435 4441 generic.go:356] "Generic (PLEG): container finished" podID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerID="236bebe2f88ca13336b3c2a88975daccb204b49d7db961a529d1e46923929136" exitCode=60 Nov 02 22:00:03.257578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.257545 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerDied","Data":"236bebe2f88ca13336b3c2a88975daccb204b49d7db961a529d1e46923929136"} Nov 02 22:00:03.258216 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.258193 4441 scope.go:122] "RemoveContainer" containerID="236bebe2f88ca13336b3c2a88975daccb204b49d7db961a529d1e46923929136" Nov 02 22:00:03.259947 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.259928 4441 generic.go:356] "Generic (PLEG): container finished" podID="05a5c852-2c53-4e74-babf-0f702210717f" containerID="8660baa7d27dc9594041c0dfa37f62be5000a68cf8bd42b659f552bc63ed355e" exitCode=60 Nov 02 22:00:03.260055 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.260033 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerDied","Data":"8660baa7d27dc9594041c0dfa37f62be5000a68cf8bd42b659f552bc63ed355e"} Nov 02 22:00:03.260422 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.260404 4441 scope.go:122] "RemoveContainer" containerID="8660baa7d27dc9594041c0dfa37f62be5000a68cf8bd42b659f552bc63ed355e" Nov 02 22:00:03.261903 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.261870 4441 generic.go:356] "Generic (PLEG): container finished" podID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerID="22559ee183442a740c20fea0ee73444f28bb7c7b0d5baee7c2e3c52203de7516" exitCode=60 Nov 02 22:00:03.262156 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.262141 4441 scope.go:122] "RemoveContainer" containerID="22559ee183442a740c20fea0ee73444f28bb7c7b0d5baee7c2e3c52203de7516" Nov 02 22:00:03.262941 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.262760 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerDied","Data":"22559ee183442a740c20fea0ee73444f28bb7c7b0d5baee7c2e3c52203de7516"} Nov 02 22:00:03.300360 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:03.300127 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" podStartSLOduration=3.300108391 podStartE2EDuration="3.300108391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.296085845 +0000 UTC m=+2832.851714206" watchObservedRunningTime="2026-11-02 22:00:03.300108391 +0000 UTC m=+2832.855736729" Nov 02 22:00:04.283312 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.283264 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerStarted","Data":"15bd782328c435be73ddbfa05df8182505de8bda52e5c9a1043d5cead818d7d9"} Nov 02 22:00:04.286072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.286042 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerStarted","Data":"cf314109607dcb0241c916bf2132a16530913f7ee9613a2b1713feae30ba5c0a"} Nov 02 22:00:04.288144 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.288116 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerStarted","Data":"8c89bdb30fd1d30916834cdba5f75c8603dabf2787f975abc4762e8b295ef32a"} Nov 02 22:00:04.290353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.290060 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerStarted","Data":"bc3a9be68d01396061f643f5bee9328282b96241f722073e3f91101572a270e1"} Nov 02 22:00:04.292212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.292185 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerStarted","Data":"7e753b1ac07dd2123e7e32749bf57792fd463e7c19b364174e66cc53900c2ebb"} Nov 02 22:00:04.294671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.294507 4441 scope.go:122] "RemoveContainer" containerID="2a06249f7d99a96720f1049fba0842663fb8fc09d745a8f31e8099d25587595f" Nov 02 22:00:04.294809 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:04.294783 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw_odh-ai-gateway-infra(7ae81538-0e22-463d-9394-f45701065e72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" podUID="7ae81538-0e22-463d-9394-f45701065e72" Nov 02 22:00:04.296646 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.296621 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerStarted","Data":"5bedde2d62dbd84af38e09b0656782d1324e5141051c1a9c12294e5744b80b5c"} Nov 02 22:00:04.298958 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.298929 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerStarted","Data":"10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96"} Nov 02 22:00:04.299243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.299218 4441 scope.go:122] "RemoveContainer" containerID="10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96" Nov 02 22:00:04.299501 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:04.299477 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7_odh-ai-gateway-infra(53c537f1-c0de-45d3-8156-f013aa218edf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" Nov 02 22:00:04.301457 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.301434 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerStarted","Data":"51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44"} Nov 02 22:00:04.301735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.301709 4441 scope.go:122] "RemoveContainer" containerID="51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44" Nov 02 22:00:04.302003 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:04.301947 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9_odh-ai-gateway-infra(15c454cc-a1c8-4b15-98b3-f6a61c708956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" Nov 02 22:00:04.303635 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.303611 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerStarted","Data":"c860ca40b0b9920bdea42538abeb578853b52e7634ce181647345b4c399aaca9"} Nov 02 22:00:04.572197 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.572168 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:04.739072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.739035 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:04.989755 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.989719 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g"] Nov 02 22:00:04.994026 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:04.993999 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.043839 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.043717 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c4a6b9-dockercfg-fbmbv\"" Nov 02 22:00:05.044100 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.044079 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c4a6b9\"" Nov 02 22:00:05.047747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.047723 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g"] Nov 02 22:00:05.065384 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.065355 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7xj\" (UniqueName: \"kubernetes.io/projected/29544df9-e93d-43fd-93fc-dacbc538008a-kube-api-access-hq7xj\") pod \"payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.065551 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.065525 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29544df9-e93d-43fd-93fc-dacbc538008a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.159979 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.159928 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t"] Nov 02 22:00:05.165637 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.165605 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.167050 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.166945 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29544df9-e93d-43fd-93fc-dacbc538008a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.167511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.167468 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7xj\" (UniqueName: \"kubernetes.io/projected/29544df9-e93d-43fd-93fc-dacbc538008a-kube-api-access-hq7xj\") pod \"payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.167962 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.167935 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29544df9-e93d-43fd-93fc-dacbc538008a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.227112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.226474 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t"] Nov 02 22:00:05.236518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.236473 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7xj\" (UniqueName: \"kubernetes.io/projected/29544df9-e93d-43fd-93fc-dacbc538008a-kube-api-access-hq7xj\") pod \"payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.269293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.269213 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2lrd\" (UniqueName: \"kubernetes.io/projected/2f516506-1916-4436-bc71-7da08d430134-kube-api-access-g2lrd\") pod \"payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.269436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.269357 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f516506-1916-4436-bc71-7da08d430134-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.309151 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.309105 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:05.316614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.316583 4441 generic.go:356] "Generic (PLEG): container finished" podID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerID="cf314109607dcb0241c916bf2132a16530913f7ee9613a2b1713feae30ba5c0a" exitCode=60 Nov 02 22:00:05.316757 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.316653 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerDied","Data":"cf314109607dcb0241c916bf2132a16530913f7ee9613a2b1713feae30ba5c0a"} Nov 02 22:00:05.316757 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.316721 4441 scope.go:122] "RemoveContainer" containerID="68ab98ac4367b14e8d8c296f7002826bdcf85635cfc786fc79b624e9fac820cb" Nov 02 22:00:05.317043 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.317026 4441 scope.go:122] "RemoveContainer" containerID="cf314109607dcb0241c916bf2132a16530913f7ee9613a2b1713feae30ba5c0a" Nov 02 22:00:05.317333 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.317314 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl_odh-ai-gateway-infra(35f00966-4ae7-4e31-9df8-f0af26d4dcb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" Nov 02 22:00:05.319247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.319223 4441 generic.go:356] "Generic (PLEG): container finished" podID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerID="8c89bdb30fd1d30916834cdba5f75c8603dabf2787f975abc4762e8b295ef32a" exitCode=60 Nov 02 22:00:05.319374 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.319306 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerDied","Data":"8c89bdb30fd1d30916834cdba5f75c8603dabf2787f975abc4762e8b295ef32a"} Nov 02 22:00:05.319552 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.319536 4441 scope.go:122] "RemoveContainer" containerID="8c89bdb30fd1d30916834cdba5f75c8603dabf2787f975abc4762e8b295ef32a" Nov 02 22:00:05.319836 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.319794 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv_odh-ai-gateway-infra(a9b57b7e-4b68-434c-a360-5ce0f620a833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" Nov 02 22:00:05.321340 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.321319 4441 generic.go:356] "Generic (PLEG): container finished" podID="975396b8-867d-479b-8a00-7c6e235414f6" containerID="bc3a9be68d01396061f643f5bee9328282b96241f722073e3f91101572a270e1" exitCode=60 Nov 02 22:00:05.321437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.321407 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerDied","Data":"bc3a9be68d01396061f643f5bee9328282b96241f722073e3f91101572a270e1"} Nov 02 22:00:05.321580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.321562 4441 scope.go:122] "RemoveContainer" containerID="bc3a9be68d01396061f643f5bee9328282b96241f722073e3f91101572a270e1" Nov 02 22:00:05.321826 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.321803 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd_odh-ai-gateway-infra(975396b8-867d-479b-8a00-7c6e235414f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" podUID="975396b8-867d-479b-8a00-7c6e235414f6" Nov 02 22:00:05.326274 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.324122 4441 generic.go:356] "Generic (PLEG): container finished" podID="659c34e8-7543-443e-9600-c049cf185952" containerID="7e753b1ac07dd2123e7e32749bf57792fd463e7c19b364174e66cc53900c2ebb" exitCode=60 Nov 02 22:00:05.326274 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.324277 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerDied","Data":"7e753b1ac07dd2123e7e32749bf57792fd463e7c19b364174e66cc53900c2ebb"} Nov 02 22:00:05.326498 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.326427 4441 scope.go:122] "RemoveContainer" containerID="7e753b1ac07dd2123e7e32749bf57792fd463e7c19b364174e66cc53900c2ebb" Nov 02 22:00:05.327027 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.326747 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l_odh-ai-gateway-infra(659c34e8-7543-443e-9600-c049cf185952)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" podUID="659c34e8-7543-443e-9600-c049cf185952" Nov 02 22:00:05.327988 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.327938 4441 generic.go:356] "Generic (PLEG): container finished" podID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerID="5bedde2d62dbd84af38e09b0656782d1324e5141051c1a9c12294e5744b80b5c" exitCode=60 Nov 02 22:00:05.328120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.328100 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerDied","Data":"5bedde2d62dbd84af38e09b0656782d1324e5141051c1a9c12294e5744b80b5c"} Nov 02 22:00:05.328531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.328511 4441 scope.go:122] "RemoveContainer" containerID="5bedde2d62dbd84af38e09b0656782d1324e5141051c1a9c12294e5744b80b5c" Nov 02 22:00:05.328809 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.328783 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-hgdrg_odh-ai-gateway-infra(e7cc9c91-4155-413b-989f-4bc18e1de331)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" Nov 02 22:00:05.331175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.331152 4441 generic.go:356] "Generic (PLEG): container finished" podID="53c537f1-c0de-45d3-8156-f013aa218edf" containerID="10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96" exitCode=60 Nov 02 22:00:05.331292 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.331234 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerDied","Data":"10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96"} Nov 02 22:00:05.331476 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.331456 4441 scope.go:122] "RemoveContainer" containerID="10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96" Nov 02 22:00:05.331694 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.331659 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7_odh-ai-gateway-infra(53c537f1-c0de-45d3-8156-f013aa218edf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" Nov 02 22:00:05.333823 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.333800 4441 generic.go:356] "Generic (PLEG): container finished" podID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerID="51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44" exitCode=60 Nov 02 22:00:05.333917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.333901 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerDied","Data":"51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44"} Nov 02 22:00:05.334266 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.334248 4441 scope.go:122] "RemoveContainer" containerID="51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44" Nov 02 22:00:05.334469 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.334449 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9_odh-ai-gateway-infra(15c454cc-a1c8-4b15-98b3-f6a61c708956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" Nov 02 22:00:05.340253 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.340232 4441 generic.go:356] "Generic (PLEG): container finished" podID="05a5c852-2c53-4e74-babf-0f702210717f" containerID="c860ca40b0b9920bdea42538abeb578853b52e7634ce181647345b4c399aaca9" exitCode=60 Nov 02 22:00:05.340365 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.340345 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerDied","Data":"c860ca40b0b9920bdea42538abeb578853b52e7634ce181647345b4c399aaca9"} Nov 02 22:00:05.340560 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.340541 4441 scope.go:122] "RemoveContainer" containerID="c860ca40b0b9920bdea42538abeb578853b52e7634ce181647345b4c399aaca9" Nov 02 22:00:05.340800 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.340777 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6_odh-ai-gateway-infra(05a5c852-2c53-4e74-babf-0f702210717f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" podUID="05a5c852-2c53-4e74-babf-0f702210717f" Nov 02 22:00:05.342957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.342930 4441 generic.go:356] "Generic (PLEG): container finished" podID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerID="15bd782328c435be73ddbfa05df8182505de8bda52e5c9a1043d5cead818d7d9" exitCode=60 Nov 02 22:00:05.343122 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.343037 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerDied","Data":"15bd782328c435be73ddbfa05df8182505de8bda52e5c9a1043d5cead818d7d9"} Nov 02 22:00:05.343291 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.343263 4441 scope.go:122] "RemoveContainer" containerID="15bd782328c435be73ddbfa05df8182505de8bda52e5c9a1043d5cead818d7d9" Nov 02 22:00:05.343445 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:05.343423 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb_odh-ai-gateway-infra(7e8fe3e8-170b-4f22-922d-9c8633eb9379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" Nov 02 22:00:05.359891 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.359866 4441 scope.go:122] "RemoveContainer" containerID="e084862cfafec26fc8908a48f5a8ab8884759ff2ec486861f0f1c2732d73f270" Nov 02 22:00:05.370557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.370520 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2lrd\" (UniqueName: \"kubernetes.io/projected/2f516506-1916-4436-bc71-7da08d430134-kube-api-access-g2lrd\") pod \"payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.370683 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.370669 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f516506-1916-4436-bc71-7da08d430134-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.372944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.372917 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f516506-1916-4436-bc71-7da08d430134-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.376704 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.376683 4441 scope.go:122] "RemoveContainer" containerID="74ae9aba8eb667f96ed3db3aa9a1d9b348af5f79d32bf7c834f26e40debae40e" Nov 02 22:00:05.398163 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.395702 4441 scope.go:122] "RemoveContainer" containerID="4a1ae6cda7aa2b2c7eb063888247153efd735d9daa19234541e4d30515e132f5" Nov 02 22:00:05.421171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.420283 4441 scope.go:122] "RemoveContainer" containerID="5cfcb8c575e1330f31a99a3457d56631cda18a38647255fe236f05ab92f52d34" Nov 02 22:00:05.427937 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.427908 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2lrd\" (UniqueName: \"kubernetes.io/projected/2f516506-1916-4436-bc71-7da08d430134-kube-api-access-g2lrd\") pod \"payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.438209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.437897 4441 scope.go:122] "RemoveContainer" containerID="76f9428742afc2b75be7352828c8465fe774b087f70c986af4278d4472a9fbba" Nov 02 22:00:05.457861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.457787 4441 scope.go:122] "RemoveContainer" containerID="236bebe2f88ca13336b3c2a88975daccb204b49d7db961a529d1e46923929136" Nov 02 22:00:05.479232 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.479210 4441 scope.go:122] "RemoveContainer" containerID="8660baa7d27dc9594041c0dfa37f62be5000a68cf8bd42b659f552bc63ed355e" Nov 02 22:00:05.487016 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.486992 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:05.492593 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.492568 4441 scope.go:122] "RemoveContainer" containerID="22559ee183442a740c20fea0ee73444f28bb7c7b0d5baee7c2e3c52203de7516" Nov 02 22:00:05.698659 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:05.698619 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29544df9_e93d_43fd_93fc_dacbc538008a.slice/crio-c8aa55bbd208303a4b6719270837b30de45cfc65ecb1d7c31dead2f34dbb7b5f WatchSource:0}: Error finding container c8aa55bbd208303a4b6719270837b30de45cfc65ecb1d7c31dead2f34dbb7b5f: Status 404 returned error can't find the container with id c8aa55bbd208303a4b6719270837b30de45cfc65ecb1d7c31dead2f34dbb7b5f Nov 02 22:00:05.699276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.699253 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g"] Nov 02 22:00:05.857156 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:05.857124 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f516506_1916_4436_bc71_7da08d430134.slice/crio-4f8437c4402d15e4c269e31e37fe584d8b1a6b904b3964734c663b8c40aa6a33 WatchSource:0}: Error finding container 4f8437c4402d15e4c269e31e37fe584d8b1a6b904b3964734c663b8c40aa6a33: Status 404 returned error can't find the container with id 4f8437c4402d15e4c269e31e37fe584d8b1a6b904b3964734c663b8c40aa6a33 Nov 02 22:00:05.859214 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:05.859189 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t"] Nov 02 22:00:06.355802 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.355771 4441 scope.go:122] "RemoveContainer" containerID="c860ca40b0b9920bdea42538abeb578853b52e7634ce181647345b4c399aaca9" Nov 02 22:00:06.356333 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.356116 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6_odh-ai-gateway-infra(05a5c852-2c53-4e74-babf-0f702210717f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" podUID="05a5c852-2c53-4e74-babf-0f702210717f" Nov 02 22:00:06.357312 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.357287 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" event={"ID":"29544df9-e93d-43fd-93fc-dacbc538008a","Type":"ContainerStarted","Data":"57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7"} Nov 02 22:00:06.357448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.357337 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" event={"ID":"29544df9-e93d-43fd-93fc-dacbc538008a","Type":"ContainerStarted","Data":"c8aa55bbd208303a4b6719270837b30de45cfc65ecb1d7c31dead2f34dbb7b5f"} Nov 02 22:00:06.357448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.357359 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:06.359321 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.359300 4441 scope.go:122] "RemoveContainer" containerID="15bd782328c435be73ddbfa05df8182505de8bda52e5c9a1043d5cead818d7d9" Nov 02 22:00:06.359604 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.359584 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb_odh-ai-gateway-infra(7e8fe3e8-170b-4f22-922d-9c8633eb9379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" Nov 02 22:00:06.361213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.361193 4441 scope.go:122] "RemoveContainer" containerID="cf314109607dcb0241c916bf2132a16530913f7ee9613a2b1713feae30ba5c0a" Nov 02 22:00:06.361420 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.361398 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl_odh-ai-gateway-infra(35f00966-4ae7-4e31-9df8-f0af26d4dcb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" Nov 02 22:00:06.362945 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.362928 4441 scope.go:122] "RemoveContainer" containerID="8c89bdb30fd1d30916834cdba5f75c8603dabf2787f975abc4762e8b295ef32a" Nov 02 22:00:06.363162 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.363137 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv_odh-ai-gateway-infra(a9b57b7e-4b68-434c-a360-5ce0f620a833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" Nov 02 22:00:06.364751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.364726 4441 scope.go:122] "RemoveContainer" containerID="bc3a9be68d01396061f643f5bee9328282b96241f722073e3f91101572a270e1" Nov 02 22:00:06.364944 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.364917 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd_odh-ai-gateway-infra(975396b8-867d-479b-8a00-7c6e235414f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" podUID="975396b8-867d-479b-8a00-7c6e235414f6" Nov 02 22:00:06.366351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.366329 4441 scope.go:122] "RemoveContainer" containerID="7e753b1ac07dd2123e7e32749bf57792fd463e7c19b364174e66cc53900c2ebb" Nov 02 22:00:06.366552 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.366528 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l_odh-ai-gateway-infra(659c34e8-7543-443e-9600-c049cf185952)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" podUID="659c34e8-7543-443e-9600-c049cf185952" Nov 02 22:00:06.367793 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.367768 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" event={"ID":"2f516506-1916-4436-bc71-7da08d430134","Type":"ContainerStarted","Data":"92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62"} Nov 02 22:00:06.367864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.367819 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" event={"ID":"2f516506-1916-4436-bc71-7da08d430134","Type":"ContainerStarted","Data":"4f8437c4402d15e4c269e31e37fe584d8b1a6b904b3964734c663b8c40aa6a33"} Nov 02 22:00:06.367864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.367850 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:06.369604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.369590 4441 scope.go:122] "RemoveContainer" containerID="5bedde2d62dbd84af38e09b0656782d1324e5141051c1a9c12294e5744b80b5c" Nov 02 22:00:06.369774 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.369753 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-hgdrg_odh-ai-gateway-infra(e7cc9c91-4155-413b-989f-4bc18e1de331)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" Nov 02 22:00:06.371236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.371220 4441 scope.go:122] "RemoveContainer" containerID="10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96" Nov 02 22:00:06.371433 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.371412 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7_odh-ai-gateway-infra(53c537f1-c0de-45d3-8156-f013aa218edf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" Nov 02 22:00:06.374037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:06.374018 4441 scope.go:122] "RemoveContainer" containerID="51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44" Nov 02 22:00:06.374254 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:06.374230 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9_odh-ai-gateway-infra(15c454cc-a1c8-4b15-98b3-f6a61c708956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" Nov 02 22:00:07.058268 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:07.056279 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" podStartSLOduration=3.056259945 podStartE2EDuration="3.056259945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:07.054861443 +0000 UTC m=+2836.610489786" watchObservedRunningTime="2026-11-02 22:00:07.056259945 +0000 UTC m=+2836.611888285" Nov 02 22:00:07.377004 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:07.376859 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" podStartSLOduration=2.376845106 podStartE2EDuration="2.376845106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:07.374540405 +0000 UTC m=+2836.930168743" watchObservedRunningTime="2026-11-02 22:00:07.376845106 +0000 UTC m=+2836.932473442" Nov 02 22:00:07.625902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:07.625870 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:10.616057 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:10.616018 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:00:12.221507 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:12.221469 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:15.576753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:15.576723 4441 scope.go:122] "RemoveContainer" containerID="10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96" Nov 02 22:00:16.455131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:16.455097 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerStarted","Data":"502a004823da1f9522cf3fde22c2dedff2d39927e90e436823c1b943e4cc3010"} Nov 02 22:00:16.577464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:16.577437 4441 scope.go:122] "RemoveContainer" containerID="2a06249f7d99a96720f1049fba0842663fb8fc09d745a8f31e8099d25587595f" Nov 02 22:00:16.577903 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:16.577612 4441 scope.go:122] "RemoveContainer" containerID="8c89bdb30fd1d30916834cdba5f75c8603dabf2787f975abc4762e8b295ef32a" Nov 02 22:00:16.577903 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:16.577787 4441 scope.go:122] "RemoveContainer" containerID="5bedde2d62dbd84af38e09b0656782d1324e5141051c1a9c12294e5744b80b5c" Nov 02 22:00:16.705178 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:16.705092 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:17.385376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.385351 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:00:17.385545 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.385421 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:00:17.464568 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.464537 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" event={"ID":"7ae81538-0e22-463d-9394-f45701065e72","Type":"ContainerStarted","Data":"09b98de5f937b096ed9776d5ff9496b5de16b3e3107e87ff1a20fe33e82c1949"} Nov 02 22:00:17.466666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.466646 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerStarted","Data":"d51caf1d979c6a666c6e350c2e2879f82d7cfd8981d53eb48d51b770624b1707"} Nov 02 22:00:17.468534 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.468515 4441 generic.go:356] "Generic (PLEG): container finished" podID="53c537f1-c0de-45d3-8156-f013aa218edf" containerID="502a004823da1f9522cf3fde22c2dedff2d39927e90e436823c1b943e4cc3010" exitCode=60 Nov 02 22:00:17.468621 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.468589 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerDied","Data":"502a004823da1f9522cf3fde22c2dedff2d39927e90e436823c1b943e4cc3010"} Nov 02 22:00:17.468673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.468641 4441 scope.go:122] "RemoveContainer" containerID="10652305e5a27941d1e4885db828bebff892ba07f7fb8157b3213fe791de4d96" Nov 02 22:00:17.468888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.468866 4441 scope.go:122] "RemoveContainer" containerID="502a004823da1f9522cf3fde22c2dedff2d39927e90e436823c1b943e4cc3010" Nov 02 22:00:17.471995 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:17.469100 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7_odh-ai-gateway-infra(53c537f1-c0de-45d3-8156-f013aa218edf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" Nov 02 22:00:17.471995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.470935 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerStarted","Data":"1cc9b7eacae2009f7a2ea9c12eec9c8eb967fcbb893fb14c62fca367ab182dd4"} Nov 02 22:00:17.577186 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.577155 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7"] Nov 02 22:00:17.577307 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.577280 4441 scope.go:122] "RemoveContainer" containerID="51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44" Nov 02 22:00:17.577460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.577426 4441 scope.go:122] "RemoveContainer" containerID="cf314109607dcb0241c916bf2132a16530913f7ee9613a2b1713feae30ba5c0a" Nov 02 22:00:17.932516 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:17.932429 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" podStartSLOduration=17.932415335 podStartE2EDuration="17.932415335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:17.931666748 +0000 UTC m=+2847.487295084" watchObservedRunningTime="2026-11-02 22:00:17.932415335 +0000 UTC m=+2847.488043668" Nov 02 22:00:18.485662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.485626 4441 generic.go:356] "Generic (PLEG): container finished" podID="7ae81538-0e22-463d-9394-f45701065e72" containerID="09b98de5f937b096ed9776d5ff9496b5de16b3e3107e87ff1a20fe33e82c1949" exitCode=60 Nov 02 22:00:18.485798 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.485698 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" event={"ID":"7ae81538-0e22-463d-9394-f45701065e72","Type":"ContainerDied","Data":"09b98de5f937b096ed9776d5ff9496b5de16b3e3107e87ff1a20fe33e82c1949"} Nov 02 22:00:18.485798 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.485768 4441 scope.go:122] "RemoveContainer" containerID="2a06249f7d99a96720f1049fba0842663fb8fc09d745a8f31e8099d25587595f" Nov 02 22:00:18.486279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.486255 4441 scope.go:122] "RemoveContainer" containerID="09b98de5f937b096ed9776d5ff9496b5de16b3e3107e87ff1a20fe33e82c1949" Nov 02 22:00:18.486645 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:18.486620 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw_odh-ai-gateway-infra(7ae81538-0e22-463d-9394-f45701065e72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" podUID="7ae81538-0e22-463d-9394-f45701065e72" Nov 02 22:00:18.488694 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.488671 4441 generic.go:356] "Generic (PLEG): container finished" podID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerID="d51caf1d979c6a666c6e350c2e2879f82d7cfd8981d53eb48d51b770624b1707" exitCode=60 Nov 02 22:00:18.488820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.488689 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerDied","Data":"d51caf1d979c6a666c6e350c2e2879f82d7cfd8981d53eb48d51b770624b1707"} Nov 02 22:00:18.489123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.489102 4441 scope.go:122] "RemoveContainer" containerID="d51caf1d979c6a666c6e350c2e2879f82d7cfd8981d53eb48d51b770624b1707" Nov 02 22:00:18.489454 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:18.489430 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-hgdrg_odh-ai-gateway-infra(e7cc9c91-4155-413b-989f-4bc18e1de331)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" Nov 02 22:00:18.493872 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.493850 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerStarted","Data":"465693772ef33efcac10fee97572088873516456903a766eae302e67989e8f0e"} Nov 02 22:00:18.494293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.494276 4441 scope.go:122] "RemoveContainer" containerID="465693772ef33efcac10fee97572088873516456903a766eae302e67989e8f0e" Nov 02 22:00:18.494697 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:18.494578 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9_odh-ai-gateway-infra(15c454cc-a1c8-4b15-98b3-f6a61c708956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" Nov 02 22:00:18.499505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.497139 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerStarted","Data":"a02238cc81b6dd03115321bdb1b833e85600c1f06da4fcc57e329a5050232be8"} Nov 02 22:00:18.499505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.497474 4441 scope.go:122] "RemoveContainer" containerID="a02238cc81b6dd03115321bdb1b833e85600c1f06da4fcc57e329a5050232be8" Nov 02 22:00:18.499505 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:18.497751 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl_odh-ai-gateway-infra(35f00966-4ae7-4e31-9df8-f0af26d4dcb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" Nov 02 22:00:18.500607 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.500564 4441 generic.go:356] "Generic (PLEG): container finished" podID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerID="1cc9b7eacae2009f7a2ea9c12eec9c8eb967fcbb893fb14c62fca367ab182dd4" exitCode=60 Nov 02 22:00:18.500711 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.500694 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerDied","Data":"1cc9b7eacae2009f7a2ea9c12eec9c8eb967fcbb893fb14c62fca367ab182dd4"} Nov 02 22:00:18.501050 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.501030 4441 scope.go:122] "RemoveContainer" containerID="1cc9b7eacae2009f7a2ea9c12eec9c8eb967fcbb893fb14c62fca367ab182dd4" Nov 02 22:00:18.502437 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:18.502408 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv_odh-ai-gateway-infra(a9b57b7e-4b68-434c-a360-5ce0f620a833)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" Nov 02 22:00:18.590379 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.590345 4441 scope.go:122] "RemoveContainer" containerID="bc3a9be68d01396061f643f5bee9328282b96241f722073e3f91101572a270e1" Nov 02 22:00:18.591385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.591358 4441 scope.go:122] "RemoveContainer" containerID="7e753b1ac07dd2123e7e32749bf57792fd463e7c19b364174e66cc53900c2ebb" Nov 02 22:00:18.591873 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.591853 4441 scope.go:122] "RemoveContainer" containerID="c860ca40b0b9920bdea42538abeb578853b52e7634ce181647345b4c399aaca9" Nov 02 22:00:18.592264 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.592241 4441 scope.go:122] "RemoveContainer" containerID="15bd782328c435be73ddbfa05df8182505de8bda52e5c9a1043d5cead818d7d9" Nov 02 22:00:18.593778 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.593760 4441 scope.go:122] "RemoveContainer" containerID="5bedde2d62dbd84af38e09b0656782d1324e5141051c1a9c12294e5744b80b5c" Nov 02 22:00:18.660307 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.660196 4441 scope.go:122] "RemoveContainer" containerID="8c89bdb30fd1d30916834cdba5f75c8603dabf2787f975abc4762e8b295ef32a" Nov 02 22:00:18.717943 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.717920 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:18.727796 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.727770 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv"] Nov 02 22:00:18.834657 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.834622 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c537f1-c0de-45d3-8156-f013aa218edf-kube-api-access-m278t\" (UniqueName: \"kubernetes.io/projected/53c537f1-c0de-45d3-8156-f013aa218edf-kube-api-access-m278t\") pod \"53c537f1-c0de-45d3-8156-f013aa218edf\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " Nov 02 22:00:18.834836 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.834719 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c537f1-c0de-45d3-8156-f013aa218edf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c537f1-c0de-45d3-8156-f013aa218edf-maas-api-service-ca\") pod \"53c537f1-c0de-45d3-8156-f013aa218edf\" (UID: \"53c537f1-c0de-45d3-8156-f013aa218edf\") " Nov 02 22:00:18.835106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.835084 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c537f1-c0de-45d3-8156-f013aa218edf-maas-api-service-ca" pod "53c537f1-c0de-45d3-8156-f013aa218edf" (UID: "53c537f1-c0de-45d3-8156-f013aa218edf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:18.835318 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.835306 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c537f1-c0de-45d3-8156-f013aa218edf-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:18.871233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.871203 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c537f1-c0de-45d3-8156-f013aa218edf-kube-api-access-m278t" pod "53c537f1-c0de-45d3-8156-f013aa218edf" (UID: "53c537f1-c0de-45d3-8156-f013aa218edf"). InnerVolumeSpecName "kube-api-access-m278t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:18.936354 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.936329 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m278t\" (UniqueName: \"kubernetes.io/projected/53c537f1-c0de-45d3-8156-f013aa218edf-kube-api-access-m278t\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:18.943276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:18.943250 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw"] Nov 02 22:00:19.054676 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.054646 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg"] Nov 02 22:00:19.514359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.514325 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" event={"ID":"53c537f1-c0de-45d3-8156-f013aa218edf","Type":"ContainerDied","Data":"f0bc585d9ca7d22cc9c51491be6baad70e86ee167ece8fa352df2bdff6d00d55"} Nov 02 22:00:19.514492 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.514399 4441 scope.go:122] "RemoveContainer" containerID="502a004823da1f9522cf3fde22c2dedff2d39927e90e436823c1b943e4cc3010" Nov 02 22:00:19.514858 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.514826 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7" Nov 02 22:00:19.517558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.517474 4441 generic.go:356] "Generic (PLEG): container finished" podID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerID="465693772ef33efcac10fee97572088873516456903a766eae302e67989e8f0e" exitCode=60 Nov 02 22:00:19.517675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.517575 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerDied","Data":"465693772ef33efcac10fee97572088873516456903a766eae302e67989e8f0e"} Nov 02 22:00:19.518302 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.518061 4441 scope.go:122] "RemoveContainer" containerID="465693772ef33efcac10fee97572088873516456903a766eae302e67989e8f0e" Nov 02 22:00:19.518440 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:19.518414 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9_odh-ai-gateway-infra(15c454cc-a1c8-4b15-98b3-f6a61c708956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" Nov 02 22:00:19.521130 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.521108 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerStarted","Data":"dcd5ac2b6b33d73bc54f6fc670e31a85e299000f65700e0182db591b5a5709db"} Nov 02 22:00:19.523767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.523744 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerStarted","Data":"19c86894a7302ebddb358151e8f7048c452c0af04b16bba95d0546b81c2b8d63"} Nov 02 22:00:19.526481 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.526457 4441 generic.go:356] "Generic (PLEG): container finished" podID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerID="a02238cc81b6dd03115321bdb1b833e85600c1f06da4fcc57e329a5050232be8" exitCode=60 Nov 02 22:00:19.526570 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.526534 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerDied","Data":"a02238cc81b6dd03115321bdb1b833e85600c1f06da4fcc57e329a5050232be8"} Nov 02 22:00:19.527056 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.526954 4441 scope.go:122] "RemoveContainer" containerID="a02238cc81b6dd03115321bdb1b833e85600c1f06da4fcc57e329a5050232be8" Nov 02 22:00:19.527277 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:19.527250 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl_odh-ai-gateway-infra(35f00966-4ae7-4e31-9df8-f0af26d4dcb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" Nov 02 22:00:19.529803 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.529779 4441 scope.go:122] "RemoveContainer" containerID="51e0df9902ab2cfd90e9d48b6f8c89b98f9434cf04c655c36969916c7848ee44" Nov 02 22:00:19.531638 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.531611 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerStarted","Data":"18dd6d04109d5f6ab8dc3efe57bc56c012ed7633ab7c06eb55fedfaa381e0e6f"} Nov 02 22:00:19.533782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.533758 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerStarted","Data":"c27462d7368dc22f00877b461a8c6bdf43017a898bac2fa8fcc1fd035acf818c"} Nov 02 22:00:19.544866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.544846 4441 scope.go:122] "RemoveContainer" containerID="cf314109607dcb0241c916bf2132a16530913f7ee9613a2b1713feae30ba5c0a" Nov 02 22:00:19.725640 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.719904 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl"] Nov 02 22:00:19.822530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.822506 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9"] Nov 02 22:00:19.941504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.941476 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:19.943654 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.943629 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7"] Nov 02 22:00:19.946072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.946050 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:19.947855 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.947830 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894280-g6xw7"] Nov 02 22:00:19.949152 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:19.949134 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:20.057279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057255 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cc9c91-4155-413b-989f-4bc18e1de331-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cc9c91-4155-413b-989f-4bc18e1de331-maas-api-service-ca\") pod \"e7cc9c91-4155-413b-989f-4bc18e1de331\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " Nov 02 22:00:20.057408 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057303 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cc9c91-4155-413b-989f-4bc18e1de331-kube-api-access-cm66k\" (UniqueName: \"kubernetes.io/projected/e7cc9c91-4155-413b-989f-4bc18e1de331-kube-api-access-cm66k\") pod \"e7cc9c91-4155-413b-989f-4bc18e1de331\" (UID: \"e7cc9c91-4155-413b-989f-4bc18e1de331\") " Nov 02 22:00:20.057408 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057359 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b57b7e-4b68-434c-a360-5ce0f620a833-kube-api-access-27x9c\" (UniqueName: \"kubernetes.io/projected/a9b57b7e-4b68-434c-a360-5ce0f620a833-kube-api-access-27x9c\") pod \"a9b57b7e-4b68-434c-a360-5ce0f620a833\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " Nov 02 22:00:20.057509 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057413 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b57b7e-4b68-434c-a360-5ce0f620a833-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b57b7e-4b68-434c-a360-5ce0f620a833-maas-api-service-ca\") pod \"a9b57b7e-4b68-434c-a360-5ce0f620a833\" (UID: \"a9b57b7e-4b68-434c-a360-5ce0f620a833\") " Nov 02 22:00:20.057509 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057486 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae81538-0e22-463d-9394-f45701065e72-kube-api-access-kjw4j\" (UniqueName: \"kubernetes.io/projected/7ae81538-0e22-463d-9394-f45701065e72-kube-api-access-kjw4j\") pod \"7ae81538-0e22-463d-9394-f45701065e72\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " Nov 02 22:00:20.057615 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057591 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae81538-0e22-463d-9394-f45701065e72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae81538-0e22-463d-9394-f45701065e72-maas-api-service-ca\") pod \"7ae81538-0e22-463d-9394-f45701065e72\" (UID: \"7ae81538-0e22-463d-9394-f45701065e72\") " Nov 02 22:00:20.057673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057612 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cc9c91-4155-413b-989f-4bc18e1de331-maas-api-service-ca" pod "e7cc9c91-4155-413b-989f-4bc18e1de331" (UID: "e7cc9c91-4155-413b-989f-4bc18e1de331"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.057821 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.057801 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b57b7e-4b68-434c-a360-5ce0f620a833-maas-api-service-ca" pod "a9b57b7e-4b68-434c-a360-5ce0f620a833" (UID: "a9b57b7e-4b68-434c-a360-5ce0f620a833"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.058037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.058009 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae81538-0e22-463d-9394-f45701065e72-maas-api-service-ca" pod "7ae81538-0e22-463d-9394-f45701065e72" (UID: "7ae81538-0e22-463d-9394-f45701065e72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.058154 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.058136 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cc9c91-4155-413b-989f-4bc18e1de331-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.058208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.058161 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b57b7e-4b68-434c-a360-5ce0f620a833-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.058208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.058176 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae81538-0e22-463d-9394-f45701065e72-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.059541 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.059512 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae81538-0e22-463d-9394-f45701065e72-kube-api-access-kjw4j" pod "7ae81538-0e22-463d-9394-f45701065e72" (UID: "7ae81538-0e22-463d-9394-f45701065e72"). InnerVolumeSpecName "kube-api-access-kjw4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.059541 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.059530 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cc9c91-4155-413b-989f-4bc18e1de331-kube-api-access-cm66k" pod "e7cc9c91-4155-413b-989f-4bc18e1de331" (UID: "e7cc9c91-4155-413b-989f-4bc18e1de331"). InnerVolumeSpecName "kube-api-access-cm66k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.059687 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.059553 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b57b7e-4b68-434c-a360-5ce0f620a833-kube-api-access-27x9c" pod "a9b57b7e-4b68-434c-a360-5ce0f620a833" (UID: "a9b57b7e-4b68-434c-a360-5ce0f620a833"). InnerVolumeSpecName "kube-api-access-27x9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.159559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.159531 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjw4j\" (UniqueName: \"kubernetes.io/projected/7ae81538-0e22-463d-9394-f45701065e72-kube-api-access-kjw4j\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.159559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.159559 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm66k\" (UniqueName: \"kubernetes.io/projected/e7cc9c91-4155-413b-989f-4bc18e1de331-kube-api-access-cm66k\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.159720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.159569 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27x9c\" (UniqueName: \"kubernetes.io/projected/a9b57b7e-4b68-434c-a360-5ce0f620a833-kube-api-access-27x9c\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.544268 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.544237 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" Nov 02 22:00:20.544384 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.544287 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv" event={"ID":"a9b57b7e-4b68-434c-a360-5ce0f620a833","Type":"ContainerDied","Data":"ec40f7f3631298630ceb3a5a84efc09dacd717122f618b22977c7b944c9b254f"} Nov 02 22:00:20.544384 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.544352 4441 scope.go:122] "RemoveContainer" containerID="1cc9b7eacae2009f7a2ea9c12eec9c8eb967fcbb893fb14c62fca367ab182dd4" Nov 02 22:00:20.546704 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.546677 4441 generic.go:356] "Generic (PLEG): container finished" podID="975396b8-867d-479b-8a00-7c6e235414f6" containerID="18dd6d04109d5f6ab8dc3efe57bc56c012ed7633ab7c06eb55fedfaa381e0e6f" exitCode=60 Nov 02 22:00:20.546848 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.546744 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerDied","Data":"18dd6d04109d5f6ab8dc3efe57bc56c012ed7633ab7c06eb55fedfaa381e0e6f"} Nov 02 22:00:20.547110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.547094 4441 scope.go:122] "RemoveContainer" containerID="18dd6d04109d5f6ab8dc3efe57bc56c012ed7633ab7c06eb55fedfaa381e0e6f" Nov 02 22:00:20.547404 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:20.547380 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd_odh-ai-gateway-infra(975396b8-867d-479b-8a00-7c6e235414f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" podUID="975396b8-867d-479b-8a00-7c6e235414f6" Nov 02 22:00:20.553501 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.553479 4441 generic.go:356] "Generic (PLEG): container finished" podID="659c34e8-7543-443e-9600-c049cf185952" containerID="c27462d7368dc22f00877b461a8c6bdf43017a898bac2fa8fcc1fd035acf818c" exitCode=60 Nov 02 22:00:20.553606 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.553556 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerDied","Data":"c27462d7368dc22f00877b461a8c6bdf43017a898bac2fa8fcc1fd035acf818c"} Nov 02 22:00:20.554247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.554228 4441 scope.go:122] "RemoveContainer" containerID="c27462d7368dc22f00877b461a8c6bdf43017a898bac2fa8fcc1fd035acf818c" Nov 02 22:00:20.554608 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:20.554581 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l_odh-ai-gateway-infra(659c34e8-7543-443e-9600-c049cf185952)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" podUID="659c34e8-7543-443e-9600-c049cf185952" Nov 02 22:00:20.555611 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.555589 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" event={"ID":"7ae81538-0e22-463d-9394-f45701065e72","Type":"ContainerDied","Data":"e3736e0cf88c13cf21fd1ce20c8d2039164b7c919a0edd06925203ac287ccc73"} Nov 02 22:00:20.555695 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.555628 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw" Nov 02 22:00:20.556825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.556798 4441 scope.go:122] "RemoveContainer" containerID="bc3a9be68d01396061f643f5bee9328282b96241f722073e3f91101572a270e1" Nov 02 22:00:20.557037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.557017 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" event={"ID":"e7cc9c91-4155-413b-989f-4bc18e1de331","Type":"ContainerDied","Data":"b6e7a91cd13ea3efd26c68ee47a7c217073a3fcac1bcb01ffe52e0a3fb2fd7bf"} Nov 02 22:00:20.557144 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.557089 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg" Nov 02 22:00:20.567078 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.567053 4441 generic.go:356] "Generic (PLEG): container finished" podID="05a5c852-2c53-4e74-babf-0f702210717f" containerID="dcd5ac2b6b33d73bc54f6fc670e31a85e299000f65700e0182db591b5a5709db" exitCode=60 Nov 02 22:00:20.567175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.567158 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerDied","Data":"dcd5ac2b6b33d73bc54f6fc670e31a85e299000f65700e0182db591b5a5709db"} Nov 02 22:00:20.567511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.567494 4441 scope.go:122] "RemoveContainer" containerID="dcd5ac2b6b33d73bc54f6fc670e31a85e299000f65700e0182db591b5a5709db" Nov 02 22:00:20.567775 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:20.567753 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6_odh-ai-gateway-infra(05a5c852-2c53-4e74-babf-0f702210717f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" podUID="05a5c852-2c53-4e74-babf-0f702210717f" Nov 02 22:00:20.570152 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.570133 4441 generic.go:356] "Generic (PLEG): container finished" podID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerID="19c86894a7302ebddb358151e8f7048c452c0af04b16bba95d0546b81c2b8d63" exitCode=60 Nov 02 22:00:20.570222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.570161 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerDied","Data":"19c86894a7302ebddb358151e8f7048c452c0af04b16bba95d0546b81c2b8d63"} Nov 02 22:00:20.570429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.570412 4441 scope.go:122] "RemoveContainer" containerID="19c86894a7302ebddb358151e8f7048c452c0af04b16bba95d0546b81c2b8d63" Nov 02 22:00:20.570651 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:20.570633 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb_odh-ai-gateway-infra(7e8fe3e8-170b-4f22-922d-9c8633eb9379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" Nov 02 22:00:20.589164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.587672 4441 scope.go:122] "RemoveContainer" containerID="7e753b1ac07dd2123e7e32749bf57792fd463e7c19b364174e66cc53900c2ebb" Nov 02 22:00:20.591238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.591207 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" path="/var/lib/kubelet/pods/53c537f1-c0de-45d3-8156-f013aa218edf/volumes" Nov 02 22:00:20.612464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.612439 4441 scope.go:122] "RemoveContainer" containerID="09b98de5f937b096ed9776d5ff9496b5de16b3e3107e87ff1a20fe33e82c1949" Nov 02 22:00:20.641343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.641087 4441 scope.go:122] "RemoveContainer" containerID="d51caf1d979c6a666c6e350c2e2879f82d7cfd8981d53eb48d51b770624b1707" Nov 02 22:00:20.662673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.662649 4441 scope.go:122] "RemoveContainer" containerID="c860ca40b0b9920bdea42538abeb578853b52e7634ce181647345b4c399aaca9" Nov 02 22:00:20.729800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.728181 4441 scope.go:122] "RemoveContainer" containerID="15bd782328c435be73ddbfa05df8182505de8bda52e5c9a1043d5cead818d7d9" Nov 02 22:00:20.808072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.808049 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:20.834549 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.834519 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:20.847931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.847906 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd"] Nov 02 22:00:20.868496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.868468 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-kube-api-access-dq96k\" (UniqueName: \"kubernetes.io/projected/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-kube-api-access-dq96k\") pod \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " Nov 02 22:00:20.868622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.868599 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-maas-api-service-ca\") pod \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\" (UID: \"35f00966-4ae7-4e31-9df8-f0af26d4dcb8\") " Nov 02 22:00:20.869034 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.868959 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-maas-api-service-ca" pod "35f00966-4ae7-4e31-9df8-f0af26d4dcb8" (UID: "35f00966-4ae7-4e31-9df8-f0af26d4dcb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.870775 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.870751 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-kube-api-access-dq96k" pod "35f00966-4ae7-4e31-9df8-f0af26d4dcb8" (UID: "35f00966-4ae7-4e31-9df8-f0af26d4dcb8"). InnerVolumeSpecName "kube-api-access-dq96k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.970248 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.970215 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c454cc-a1c8-4b15-98b3-f6a61c708956-kube-api-access-pq4pb\" (UniqueName: \"kubernetes.io/projected/15c454cc-a1c8-4b15-98b3-f6a61c708956-kube-api-access-pq4pb\") pod \"15c454cc-a1c8-4b15-98b3-f6a61c708956\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " Nov 02 22:00:20.970735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.970267 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c454cc-a1c8-4b15-98b3-f6a61c708956-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c454cc-a1c8-4b15-98b3-f6a61c708956-maas-api-service-ca\") pod \"15c454cc-a1c8-4b15-98b3-f6a61c708956\" (UID: \"15c454cc-a1c8-4b15-98b3-f6a61c708956\") " Nov 02 22:00:20.970735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.970660 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c454cc-a1c8-4b15-98b3-f6a61c708956-maas-api-service-ca" pod "15c454cc-a1c8-4b15-98b3-f6a61c708956" (UID: "15c454cc-a1c8-4b15-98b3-f6a61c708956"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.970832 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.970816 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.970870 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.970836 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq96k\" (UniqueName: \"kubernetes.io/projected/35f00966-4ae7-4e31-9df8-f0af26d4dcb8-kube-api-access-dq96k\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.970870 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.970847 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c454cc-a1c8-4b15-98b3-f6a61c708956-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.972470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:20.972446 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c454cc-a1c8-4b15-98b3-f6a61c708956-kube-api-access-pq4pb" pod "15c454cc-a1c8-4b15-98b3-f6a61c708956" (UID: "15c454cc-a1c8-4b15-98b3-f6a61c708956"). InnerVolumeSpecName "kube-api-access-pq4pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:21.066906 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.066868 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l"] Nov 02 22:00:21.072046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.071958 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq4pb\" (UniqueName: \"kubernetes.io/projected/15c454cc-a1c8-4b15-98b3-f6a61c708956-kube-api-access-pq4pb\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:21.103618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.103574 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg"] Nov 02 22:00:21.109711 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.109682 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-hgdrg"] Nov 02 22:00:21.310807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.310771 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:21.380984 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.380871 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv"] Nov 02 22:00:21.387362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.387330 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3566f9-29894280-55fnv"] Nov 02 22:00:21.391278 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.391253 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6"] Nov 02 22:00:21.580160 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.580129 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" Nov 02 22:00:21.580355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.580127 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl" event={"ID":"35f00966-4ae7-4e31-9df8-f0af26d4dcb8","Type":"ContainerDied","Data":"d7b74e07aa2e4e6ab19657775a86a01f3eeb505084107a330cf22a7379bc5dfd"} Nov 02 22:00:21.580355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.580293 4441 scope.go:122] "RemoveContainer" containerID="a02238cc81b6dd03115321bdb1b833e85600c1f06da4fcc57e329a5050232be8" Nov 02 22:00:21.590738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.590713 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" event={"ID":"15c454cc-a1c8-4b15-98b3-f6a61c708956","Type":"ContainerDied","Data":"92ba0597e50dc7c1ab41d87dd79b1483133fdef371f621d330da2efe67e2e764"} Nov 02 22:00:21.591100 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.591079 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9" Nov 02 22:00:21.595007 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.594835 4441 scope.go:122] "RemoveContainer" containerID="465693772ef33efcac10fee97572088873516456903a766eae302e67989e8f0e" Nov 02 22:00:21.601296 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.601270 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb"] Nov 02 22:00:21.785837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.784547 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw"] Nov 02 22:00:21.788763 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.788731 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d4f799-29894280-sw7sw"] Nov 02 22:00:21.867451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.867367 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:21.935038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.934938 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:21.938929 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.938906 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:21.943081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.943063 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:21.985511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.985477 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975396b8-867d-479b-8a00-7c6e235414f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975396b8-867d-479b-8a00-7c6e235414f6-maas-api-service-ca\") pod \"975396b8-867d-479b-8a00-7c6e235414f6\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " Nov 02 22:00:21.985895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.985538 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975396b8-867d-479b-8a00-7c6e235414f6-kube-api-access-z2fw6\" (UniqueName: \"kubernetes.io/projected/975396b8-867d-479b-8a00-7c6e235414f6-kube-api-access-z2fw6\") pod \"975396b8-867d-479b-8a00-7c6e235414f6\" (UID: \"975396b8-867d-479b-8a00-7c6e235414f6\") " Nov 02 22:00:21.985895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.985847 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975396b8-867d-479b-8a00-7c6e235414f6-maas-api-service-ca" pod "975396b8-867d-479b-8a00-7c6e235414f6" (UID: "975396b8-867d-479b-8a00-7c6e235414f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:21.986055 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.986040 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975396b8-867d-479b-8a00-7c6e235414f6-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:21.987663 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:21.987642 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975396b8-867d-479b-8a00-7c6e235414f6-kube-api-access-z2fw6" pod "975396b8-867d-479b-8a00-7c6e235414f6" (UID: "975396b8-867d-479b-8a00-7c6e235414f6"). InnerVolumeSpecName "kube-api-access-z2fw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:22.086715 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.086675 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8fe3e8-170b-4f22-922d-9c8633eb9379-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fe3e8-170b-4f22-922d-9c8633eb9379-maas-api-service-ca\") pod \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " Nov 02 22:00:22.086893 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.086723 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659c34e8-7543-443e-9600-c049cf185952-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c34e8-7543-443e-9600-c049cf185952-maas-api-service-ca\") pod \"659c34e8-7543-443e-9600-c049cf185952\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " Nov 02 22:00:22.086893 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.086759 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659c34e8-7543-443e-9600-c049cf185952-kube-api-access-hr7hx\" (UniqueName: \"kubernetes.io/projected/659c34e8-7543-443e-9600-c049cf185952-kube-api-access-hr7hx\") pod \"659c34e8-7543-443e-9600-c049cf185952\" (UID: \"659c34e8-7543-443e-9600-c049cf185952\") " Nov 02 22:00:22.086893 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.086889 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a5c852-2c53-4e74-babf-0f702210717f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5c852-2c53-4e74-babf-0f702210717f-maas-api-service-ca\") pod \"05a5c852-2c53-4e74-babf-0f702210717f\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " Nov 02 22:00:22.087092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.086953 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8fe3e8-170b-4f22-922d-9c8633eb9379-kube-api-access-zmnkq\" (UniqueName: \"kubernetes.io/projected/7e8fe3e8-170b-4f22-922d-9c8633eb9379-kube-api-access-zmnkq\") pod \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\" (UID: \"7e8fe3e8-170b-4f22-922d-9c8633eb9379\") " Nov 02 22:00:22.087092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087044 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a5c852-2c53-4e74-babf-0f702210717f-kube-api-access-z6wnd\" (UniqueName: \"kubernetes.io/projected/05a5c852-2c53-4e74-babf-0f702210717f-kube-api-access-z6wnd\") pod \"05a5c852-2c53-4e74-babf-0f702210717f\" (UID: \"05a5c852-2c53-4e74-babf-0f702210717f\") " Nov 02 22:00:22.087247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087160 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659c34e8-7543-443e-9600-c049cf185952-maas-api-service-ca" pod "659c34e8-7543-443e-9600-c049cf185952" (UID: "659c34e8-7543-443e-9600-c049cf185952"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:22.087343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087318 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a5c852-2c53-4e74-babf-0f702210717f-maas-api-service-ca" pod "05a5c852-2c53-4e74-babf-0f702210717f" (UID: "05a5c852-2c53-4e74-babf-0f702210717f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:22.087402 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087356 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8fe3e8-170b-4f22-922d-9c8633eb9379-maas-api-service-ca" pod "7e8fe3e8-170b-4f22-922d-9c8633eb9379" (UID: "7e8fe3e8-170b-4f22-922d-9c8633eb9379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:22.087624 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087589 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fe3e8-170b-4f22-922d-9c8633eb9379-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:22.087624 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087617 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c34e8-7543-443e-9600-c049cf185952-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:22.087839 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087633 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2fw6\" (UniqueName: \"kubernetes.io/projected/975396b8-867d-479b-8a00-7c6e235414f6-kube-api-access-z2fw6\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:22.087839 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.087646 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5c852-2c53-4e74-babf-0f702210717f-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:22.089527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.089481 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8fe3e8-170b-4f22-922d-9c8633eb9379-kube-api-access-zmnkq" pod "7e8fe3e8-170b-4f22-922d-9c8633eb9379" (UID: "7e8fe3e8-170b-4f22-922d-9c8633eb9379"). InnerVolumeSpecName "kube-api-access-zmnkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:22.089634 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.089588 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659c34e8-7543-443e-9600-c049cf185952-kube-api-access-hr7hx" pod "659c34e8-7543-443e-9600-c049cf185952" (UID: "659c34e8-7543-443e-9600-c049cf185952"). InnerVolumeSpecName "kube-api-access-hr7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:22.089702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.089682 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a5c852-2c53-4e74-babf-0f702210717f-kube-api-access-z6wnd" pod "05a5c852-2c53-4e74-babf-0f702210717f" (UID: "05a5c852-2c53-4e74-babf-0f702210717f"). InnerVolumeSpecName "kube-api-access-z6wnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:22.188857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.188772 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmnkq\" (UniqueName: \"kubernetes.io/projected/7e8fe3e8-170b-4f22-922d-9c8633eb9379-kube-api-access-zmnkq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:22.188857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.188811 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6wnd\" (UniqueName: \"kubernetes.io/projected/05a5c852-2c53-4e74-babf-0f702210717f-kube-api-access-z6wnd\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:22.188857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.188826 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr7hx\" (UniqueName: \"kubernetes.io/projected/659c34e8-7543-443e-9600-c049cf185952-kube-api-access-hr7hx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:22.380124 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.380090 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl"] Nov 02 22:00:22.388772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.388733 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-70d237-29894280-g6jpl"] Nov 02 22:00:22.544070 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.543953 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9"] Nov 02 22:00:22.547836 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.547811 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894280-b8kg9"] Nov 02 22:00:22.585005 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.584960 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" path="/var/lib/kubelet/pods/15c454cc-a1c8-4b15-98b3-f6a61c708956/volumes" Nov 02 22:00:22.585434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.585421 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" path="/var/lib/kubelet/pods/35f00966-4ae7-4e31-9df8-f0af26d4dcb8/volumes" Nov 02 22:00:22.585791 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.585779 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae81538-0e22-463d-9394-f45701065e72" path="/var/lib/kubelet/pods/7ae81538-0e22-463d-9394-f45701065e72/volumes" Nov 02 22:00:22.586482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.586467 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" path="/var/lib/kubelet/pods/a9b57b7e-4b68-434c-a360-5ce0f620a833/volumes" Nov 02 22:00:22.586841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.586828 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" path="/var/lib/kubelet/pods/e7cc9c91-4155-413b-989f-4bc18e1de331/volumes" Nov 02 22:00:22.617990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.609777 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" event={"ID":"975396b8-867d-479b-8a00-7c6e235414f6","Type":"ContainerDied","Data":"d50fc2cfca8acf9f5c977587ea880b4085901f8138ec5fa5eaa13f191213c82e"} Nov 02 22:00:22.617990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.609798 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd" Nov 02 22:00:22.617990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.609853 4441 scope.go:122] "RemoveContainer" containerID="18dd6d04109d5f6ab8dc3efe57bc56c012ed7633ab7c06eb55fedfaa381e0e6f" Nov 02 22:00:22.617990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.612124 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" event={"ID":"659c34e8-7543-443e-9600-c049cf185952","Type":"ContainerDied","Data":"a98161d0935b8bfef32e80fb76f45f4e8d3230cea57167b692abff1beaef48f6"} Nov 02 22:00:22.617990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.612236 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l" Nov 02 22:00:22.617990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.616076 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" Nov 02 22:00:22.617990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.617710 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" Nov 02 22:00:22.621053 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.621020 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6" event={"ID":"05a5c852-2c53-4e74-babf-0f702210717f","Type":"ContainerDied","Data":"94ffd071296d980827b5d7225bb0d64dd2e965510cc2a813a21f2a648e0afd75"} Nov 02 22:00:22.621178 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.621063 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb" event={"ID":"7e8fe3e8-170b-4f22-922d-9c8633eb9379","Type":"ContainerDied","Data":"5c5143507d8442b1e59daec9dbe1ecec13e395c30863f3f8a695ded917fa1855"} Nov 02 22:00:22.623066 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.623047 4441 scope.go:122] "RemoveContainer" containerID="c27462d7368dc22f00877b461a8c6bdf43017a898bac2fa8fcc1fd035acf818c" Nov 02 22:00:22.634435 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.634414 4441 scope.go:122] "RemoveContainer" containerID="dcd5ac2b6b33d73bc54f6fc670e31a85e299000f65700e0182db591b5a5709db" Nov 02 22:00:22.646579 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:22.646559 4441 scope.go:122] "RemoveContainer" containerID="19c86894a7302ebddb358151e8f7048c452c0af04b16bba95d0546b81c2b8d63" Nov 02 22:00:23.076858 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.076825 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6"] Nov 02 22:00:23.082476 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.082450 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7cc1b1-29894280-226w6"] Nov 02 22:00:23.182216 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.182178 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb"] Nov 02 22:00:23.185842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.185816 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-284b8e-29894280-492sb"] Nov 02 22:00:23.287816 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.286488 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd"] Nov 02 22:00:23.292205 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.292176 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f4a242-29894280-vqcfd"] Nov 02 22:00:23.396951 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.396866 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l"] Nov 02 22:00:23.402505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:23.402477 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-686bbf-29894280-q2k7l"] Nov 02 22:00:24.586629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:24.586600 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a5c852-2c53-4e74-babf-0f702210717f" path="/var/lib/kubelet/pods/05a5c852-2c53-4e74-babf-0f702210717f/volumes" Nov 02 22:00:24.587108 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:24.587035 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659c34e8-7543-443e-9600-c049cf185952" path="/var/lib/kubelet/pods/659c34e8-7543-443e-9600-c049cf185952/volumes" Nov 02 22:00:24.587397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:24.587383 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" path="/var/lib/kubelet/pods/7e8fe3e8-170b-4f22-922d-9c8633eb9379/volumes" Nov 02 22:00:24.588065 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:24.588050 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975396b8-867d-479b-8a00-7c6e235414f6" path="/var/lib/kubelet/pods/975396b8-867d-479b-8a00-7c6e235414f6/volumes" Nov 02 22:00:25.790853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:25.790812 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:30.387768 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:30.387715 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:34.875996 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:34.875942 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:35.072915 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:35.072869 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:36.011344 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011301 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm"] Nov 02 22:00:36.011884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011868 4441 cpu_manager.go:414] "containerMap: removing container" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.011884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011884 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011895 4441 cpu_manager.go:414] "containerMap: removing container" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011900 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011911 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011915 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011923 4441 cpu_manager.go:414] "containerMap: removing container" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011927 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011933 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011938 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011949 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011953 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011960 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011975 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011981 4441 cpu_manager.go:414] "containerMap: removing container" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011986 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011992 4441 cpu_manager.go:414] "containerMap: removing container" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.011996 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012006 4441 cpu_manager.go:414] "containerMap: removing container" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012011 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012018 4441 cpu_manager.go:414] "containerMap: removing container" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012022 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012034 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.012030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012042 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012053 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012059 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012068 4441 cpu_manager.go:414] "containerMap: removing container" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012072 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012081 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012085 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012092 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012096 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012102 4441 cpu_manager.go:414] "containerMap: removing container" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012106 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012112 4441 cpu_manager.go:414] "containerMap: removing container" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012119 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012131 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012135 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012141 4441 cpu_manager.go:414] "containerMap: removing container" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012145 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012150 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012154 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012162 4441 cpu_manager.go:414] "containerMap: removing container" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012166 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012186 4441 cpu_manager.go:414] "containerMap: removing container" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012191 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012198 4441 cpu_manager.go:414] "containerMap: removing container" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012203 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012212 4441 cpu_manager.go:414] "containerMap: removing container" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012217 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012224 4441 cpu_manager.go:414] "containerMap: removing container" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012230 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012374 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012388 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012401 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012410 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012418 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012427 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012435 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012443 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012451 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012459 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012468 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012478 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012486 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012496 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012503 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012515 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012523 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="975396b8-867d-479b-8a00-7c6e235414f6" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012534 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012540 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012546 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8fe3e8-170b-4f22-922d-9c8633eb9379" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012554 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a5c852-2c53-4e74-babf-0f702210717f" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012559 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:36.013230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.012565 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:36.018020 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.017998 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.055546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.055521 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzdf\" (UniqueName: \"kubernetes.io/projected/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-kube-api-access-8wzdf\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.055658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.055583 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.055697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.055657 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.067787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.067760 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm"] Nov 02 22:00:36.110811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.110783 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:00:36.111000 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.110908 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:00:36.114810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.114791 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:00:36.114927 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.114818 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:00:36.156849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.156797 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.156849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.156847 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.157114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.156997 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzdf\" (UniqueName: \"kubernetes.io/projected/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-kube-api-access-8wzdf\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.157556 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.157535 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.159332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.159310 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.213413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.213379 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzdf\" (UniqueName: \"kubernetes.io/projected/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-kube-api-access-8wzdf\") pod \"maas-api-revoke-sub-b04df275-gp9tm\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.333135 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.333103 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:36.712557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.712232 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm"] Nov 02 22:00:36.713073 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:36.713043 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded73a45b_72f3_4deb_91e1_6fa8490f0df8.slice/crio-17f368fb67e36d15cccb784999e1c8b9a09bfad96b3b6b72d6ce6d0bb3b3aaba WatchSource:0}: Error finding container 17f368fb67e36d15cccb784999e1c8b9a09bfad96b3b6b72d6ce6d0bb3b3aaba: Status 404 returned error can't find the container with id 17f368fb67e36d15cccb784999e1c8b9a09bfad96b3b6b72d6ce6d0bb3b3aaba Nov 02 22:00:36.753727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:36.753698 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" event={"ID":"ed73a45b-72f3-4deb-91e1-6fa8490f0df8","Type":"ContainerStarted","Data":"17f368fb67e36d15cccb784999e1c8b9a09bfad96b3b6b72d6ce6d0bb3b3aaba"} Nov 02 22:00:37.763745 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:37.763717 4441 generic.go:356] "Generic (PLEG): container finished" podID="ed73a45b-72f3-4deb-91e1-6fa8490f0df8" containerID="dd3e876a1ff2fa19714cac966af4e8f30bdc04fe1f2cf81f5cb2521d51636985" exitCode=0 Nov 02 22:00:37.764085 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:37.763755 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" event={"ID":"ed73a45b-72f3-4deb-91e1-6fa8490f0df8","Type":"ContainerDied","Data":"dd3e876a1ff2fa19714cac966af4e8f30bdc04fe1f2cf81f5cb2521d51636985"} Nov 02 22:00:38.926433 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.926406 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:38.991861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.991825 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-openshift-ca\") pod \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " Nov 02 22:00:38.992031 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.991916 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-xks-ca\") pod \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " Nov 02 22:00:38.992031 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.991979 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-kube-api-access-8wzdf\" (UniqueName: \"kubernetes.io/projected/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-kube-api-access-8wzdf\") pod \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\" (UID: \"ed73a45b-72f3-4deb-91e1-6fa8490f0df8\") " Nov 02 22:00:38.992290 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.992263 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-openshift-ca" pod "ed73a45b-72f3-4deb-91e1-6fa8490f0df8" (UID: "ed73a45b-72f3-4deb-91e1-6fa8490f0df8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:38.992559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.992533 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:38.994086 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.994058 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-xks-ca" pod "ed73a45b-72f3-4deb-91e1-6fa8490f0df8" (UID: "ed73a45b-72f3-4deb-91e1-6fa8490f0df8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:38.994236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:38.994212 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-kube-api-access-8wzdf" pod "ed73a45b-72f3-4deb-91e1-6fa8490f0df8" (UID: "ed73a45b-72f3-4deb-91e1-6fa8490f0df8"). InnerVolumeSpecName "kube-api-access-8wzdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:39.094200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:39.094162 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:39.094200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:39.094200 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzdf\" (UniqueName: \"kubernetes.io/projected/ed73a45b-72f3-4deb-91e1-6fa8490f0df8-kube-api-access-8wzdf\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:39.497328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:39.497228 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:39.784355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:39.784261 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" Nov 02 22:00:39.784355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:39.784295 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm" event={"ID":"ed73a45b-72f3-4deb-91e1-6fa8490f0df8","Type":"ContainerDied","Data":"17f368fb67e36d15cccb784999e1c8b9a09bfad96b3b6b72d6ce6d0bb3b3aaba"} Nov 02 22:00:39.784606 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:39.784388 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17f368fb67e36d15cccb784999e1c8b9a09bfad96b3b6b72d6ce6d0bb3b3aaba" Nov 02 22:00:40.620937 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:40.620899 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:00:43.426448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:43.426414 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:44.887078 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:44.887047 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:45.095493 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:45.095460 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:45.212192 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:45.212047 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:45.361273 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:45.361243 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9"] Nov 02 22:00:45.408800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:45.408768 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7cc1b1-bhln9"] Nov 02 22:00:45.720496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:45.720460 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:46.584790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:46.584756 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df88869-0b53-4e2f-8974-dcb2d5b43468" path="/var/lib/kubelet/pods/0df88869-0b53-4e2f-8974-dcb2d5b43468/volumes" Nov 02 22:00:49.965360 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:49.965283 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:51.503296 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:51.503259 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:51.624316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:51.624285 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:51.813723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:51.813693 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx"] Nov 02 22:00:51.813920 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:51.813896 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" containerID="cri-o://8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2" gracePeriod=30 Nov 02 22:00:51.854131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:51.854105 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:52.552859 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.552822 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8"] Nov 02 22:00:52.553429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553413 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:52.553484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553430 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:52.553484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553460 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ed73a45b-72f3-4deb-91e1-6fa8490f0df8" containerName="revoke-keys" Nov 02 22:00:52.553484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553466 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed73a45b-72f3-4deb-91e1-6fa8490f0df8" containerName="revoke-keys" Nov 02 22:00:52.553484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553478 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:52.553484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553483 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553503 4441 cpu_manager.go:414] "containerMap: removing container" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553507 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553521 4441 cpu_manager.go:414] "containerMap: removing container" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553527 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553617 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c34e8-7543-443e-9600-c049cf185952" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553626 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c454cc-a1c8-4b15-98b3-f6a61c708956" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553639 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed73a45b-72f3-4deb-91e1-6fa8490f0df8" containerName="revoke-keys" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553645 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b57b7e-4b68-434c-a360-5ce0f620a833" containerName="cleanup" Nov 02 22:00:52.553649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553650 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c537f1-c0de-45d3-8156-f013aa218edf" containerName="cleanup" Nov 02 22:00:52.553930 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553657 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cc9c91-4155-413b-989f-4bc18e1de331" containerName="cleanup" Nov 02 22:00:52.553930 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553662 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f00966-4ae7-4e31-9df8-f0af26d4dcb8" containerName="cleanup" Nov 02 22:00:52.553930 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.553668 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae81538-0e22-463d-9394-f45701065e72" containerName="cleanup" Nov 02 22:00:52.557202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.557187 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.577064 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.577039 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcfz\" (UniqueName: \"kubernetes.io/projected/bebaa666-c24e-4436-9943-5c91c74b6f89-kube-api-access-kwcfz\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.577189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.577152 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.577241 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.577200 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.607103 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.607074 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:00:52.607351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.607329 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:00:52.607505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.607432 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:00:52.611489 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.611465 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8"] Nov 02 22:00:52.652049 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.652019 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:00:52.678132 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.678105 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcfz\" (UniqueName: \"kubernetes.io/projected/bebaa666-c24e-4436-9943-5c91c74b6f89-kube-api-access-kwcfz\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.678250 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.678183 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.678250 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.678225 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.678860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.678842 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.680494 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.680473 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.731579 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.731553 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcfz\" (UniqueName: \"kubernetes.io/projected/bebaa666-c24e-4436-9943-5c91c74b6f89-kube-api-access-kwcfz\") pod \"maas-api-revoke-keys-e2e-inf-b-d4f799-298j8\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:52.876677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:52.876650 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:53.249048 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:00:53.249014 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebaa666_c24e_4436_9943_5c91c74b6f89.slice/crio-230e0f03faeb364a701e20d727f8254c3f81bd31706d58c8c65599f34faf2a8d WatchSource:0}: Error finding container 230e0f03faeb364a701e20d727f8254c3f81bd31706d58c8c65599f34faf2a8d: Status 404 returned error can't find the container with id 230e0f03faeb364a701e20d727f8254c3f81bd31706d58c8c65599f34faf2a8d Nov 02 22:00:53.249297 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:53.249272 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8"] Nov 02 22:00:53.893420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:53.893324 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" event={"ID":"bebaa666-c24e-4436-9943-5c91c74b6f89","Type":"ContainerStarted","Data":"143aeae8f762ac2d0ce4f1e0fbe5c222bd21ccec845007b01b421c5efa83b343"} Nov 02 22:00:53.893420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:53.893379 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" event={"ID":"bebaa666-c24e-4436-9943-5c91c74b6f89","Type":"ContainerStarted","Data":"230e0f03faeb364a701e20d727f8254c3f81bd31706d58c8c65599f34faf2a8d"} Nov 02 22:00:54.003723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:54.003654 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" podStartSLOduration=2.003637674 podStartE2EDuration="2.003637674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:53.999185665 +0000 UTC m=+2883.554814002" watchObservedRunningTime="2026-11-02 22:00:54.003637674 +0000 UTC m=+2883.559266011" Nov 02 22:00:54.653832 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:54.653789 4441 scope.go:122] "RemoveContainer" containerID="91cc12317461a80d4546d33f47500e72410a71fee95db7a82b75094847fd6f64" Nov 02 22:00:54.664466 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:54.664442 4441 scope.go:122] "RemoveContainer" containerID="d2ce5f4ad9672787ba4c56b4884baf42f50162903a5cdde607f41415aac5cfb6" Nov 02 22:00:54.672483 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:54.672462 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:54.673418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:54.673400 4441 scope.go:122] "RemoveContainer" containerID="0f324887b5fbb655fb94d8cc433224f81e08728ae7b6498a986ba85eb4d77d29" Nov 02 22:00:54.903654 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:54.903621 4441 generic.go:356] "Generic (PLEG): container finished" podID="bebaa666-c24e-4436-9943-5c91c74b6f89" containerID="143aeae8f762ac2d0ce4f1e0fbe5c222bd21ccec845007b01b421c5efa83b343" exitCode=0 Nov 02 22:00:54.904057 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:54.903676 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" event={"ID":"bebaa666-c24e-4436-9943-5c91c74b6f89","Type":"ContainerDied","Data":"143aeae8f762ac2d0ce4f1e0fbe5c222bd21ccec845007b01b421c5efa83b343"} Nov 02 22:00:56.058957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.058930 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:56.111206 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.111173 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-xks-ca\") pod \"bebaa666-c24e-4436-9943-5c91c74b6f89\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " Nov 02 22:00:56.111359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.111220 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-openshift-ca\") pod \"bebaa666-c24e-4436-9943-5c91c74b6f89\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " Nov 02 22:00:56.111359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.111284 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebaa666-c24e-4436-9943-5c91c74b6f89-kube-api-access-kwcfz\" (UniqueName: \"kubernetes.io/projected/bebaa666-c24e-4436-9943-5c91c74b6f89-kube-api-access-kwcfz\") pod \"bebaa666-c24e-4436-9943-5c91c74b6f89\" (UID: \"bebaa666-c24e-4436-9943-5c91c74b6f89\") " Nov 02 22:00:56.111614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.111584 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-openshift-ca" pod "bebaa666-c24e-4436-9943-5c91c74b6f89" (UID: "bebaa666-c24e-4436-9943-5c91c74b6f89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:56.111846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.111828 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:56.113344 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.113314 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-xks-ca" pod "bebaa666-c24e-4436-9943-5c91c74b6f89" (UID: "bebaa666-c24e-4436-9943-5c91c74b6f89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:56.113458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.113394 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebaa666-c24e-4436-9943-5c91c74b6f89-kube-api-access-kwcfz" pod "bebaa666-c24e-4436-9943-5c91c74b6f89" (UID: "bebaa666-c24e-4436-9943-5c91c74b6f89"). InnerVolumeSpecName "kube-api-access-kwcfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:56.213002 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.212879 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bebaa666-c24e-4436-9943-5c91c74b6f89-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:56.213002 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.212909 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcfz\" (UniqueName: \"kubernetes.io/projected/bebaa666-c24e-4436-9943-5c91c74b6f89-kube-api-access-kwcfz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:56.697879 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.697845 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:56.922101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.922070 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" Nov 02 22:00:56.922101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.922081 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8" event={"ID":"bebaa666-c24e-4436-9943-5c91c74b6f89","Type":"ContainerDied","Data":"230e0f03faeb364a701e20d727f8254c3f81bd31706d58c8c65599f34faf2a8d"} Nov 02 22:00:56.922293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:56.922138 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="230e0f03faeb364a701e20d727f8254c3f81bd31706d58c8c65599f34faf2a8d" Nov 02 22:00:57.092179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.092133 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 22:00:57.121036 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.120981 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-envoy\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121147 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121030 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-certs\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121147 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121099 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-podinfo\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121265 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121177 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-kube-api-access-dfvnj\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-kube-api-access-dfvnj\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121265 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121221 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-credential-socket\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121301 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-data\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121295 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-certs" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:57.121381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121358 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-socket\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121551 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121406 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-token\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-token\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121551 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121447 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03da32a5-06a1-46e4-8379-154d9524ad4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da32a5-06a1-46e4-8379-154d9524ad4d-istiod-ca-cert\") pod \"03da32a5-06a1-46e4-8379-154d9524ad4d\" (UID: \"03da32a5-06a1-46e4-8379-154d9524ad4d\") " Nov 02 22:00:57.121671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121583 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-data" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:57.121911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121885 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03da32a5-06a1-46e4-8379-154d9524ad4d-istiod-ca-cert" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:57.121911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121897 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.122111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121921 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.122111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.121947 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-socket" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:57.122111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.122026 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-credential-socket" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:57.123186 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.123153 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-envoy" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:57.123441 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.123422 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-podinfo" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:00:57.123527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.123437 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-kube-api-access-dfvnj" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "kube-api-access-dfvnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:57.123735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.123716 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-token" pod "03da32a5-06a1-46e4-8379-154d9524ad4d" (UID: "03da32a5-06a1-46e4-8379-154d9524ad4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:57.223260 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.223226 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.223260 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.223256 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfvnj\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-kube-api-access-dfvnj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.223260 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.223266 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.223554 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.223275 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.223554 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.223284 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.223554 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.223292 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da32a5-06a1-46e4-8379-154d9524ad4d-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.223554 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.223300 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da32a5-06a1-46e4-8379-154d9524ad4d-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:57.931402 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.931370 4441 generic.go:356] "Generic (PLEG): container finished" podID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerID="8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2" exitCode=0 Nov 02 22:00:57.931590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.931407 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" event={"ID":"03da32a5-06a1-46e4-8379-154d9524ad4d","Type":"ContainerDied","Data":"8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2"} Nov 02 22:00:57.931590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.931416 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" Nov 02 22:00:57.931590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.931449 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx" event={"ID":"03da32a5-06a1-46e4-8379-154d9524ad4d","Type":"ContainerDied","Data":"6bbe10f38ab01286a4d6d0f06f059a7e86d5f2c1501f8219071d145fe473c472"} Nov 02 22:00:57.931590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.931470 4441 scope.go:122] "RemoveContainer" containerID="8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2" Nov 02 22:00:57.945552 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.945518 4441 scope.go:122] "RemoveContainer" containerID="921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549" Nov 02 22:00:57.955186 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.955166 4441 scope.go:122] "RemoveContainer" containerID="8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2" Nov 02 22:00:57.955453 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:57.955432 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2\": container with ID starting with 8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2 not found: ID does not exist" containerID="8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2" Nov 02 22:00:57.955562 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.955456 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2"} err="failed to get container status \"8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2\": rpc error: code = NotFound desc = could not find container \"8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2\": container with ID starting with 8c6eff10fc8c78760dacc88c1a94b1bf0193abd5d4dce3309726af0fd2290bd2 not found: ID does not exist" Nov 02 22:00:57.955562 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.955474 4441 scope.go:122] "RemoveContainer" containerID="921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549" Nov 02 22:00:57.955726 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:00:57.955705 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549\": container with ID starting with 921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549 not found: ID does not exist" containerID="921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549" Nov 02 22:00:57.955787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:57.955735 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549"} err="failed to get container status \"921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549\": rpc error: code = NotFound desc = could not find container \"921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549\": container with ID starting with 921764c52f365e6d3947935d848aa6849e7488ef1099b109cfec85b089495549 not found: ID does not exist" Nov 02 22:00:58.089407 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:58.089369 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx"] Nov 02 22:00:58.092792 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:58.092769 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7cc1b1-openshift-default-84d569b9c-pkbzx"] Nov 02 22:00:58.585945 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:58.585912 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" path="/var/lib/kubelet/pods/03da32a5-06a1-46e4-8379-154d9524ad4d/volumes" Nov 02 22:00:59.436644 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:00:59.436612 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:03.899263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:03.899232 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:03.958365 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:03.958322 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg"] Nov 02 22:01:04.059429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:04.059397 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb"] Nov 02 22:01:04.291134 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:04.291031 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:04.417389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:04.417353 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:08.477314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:08.477273 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:08.958996 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:08.958931 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" podUID="dad61665-5a6c-4452-a0b5-32b8cd27308f" containerName="payload-processing" containerID="cri-o://434625c44f9a6326f22cf66ccbfd3325a28e3c191b1fbaec7d1109f4a0e56b59" gracePeriod=25 Nov 02 22:01:09.060189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:09.060154 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" podUID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" containerName="payload-pre-processing" containerID="cri-o://ac11e13931dbf956c7181015351243dac43cd1d0431b6d257d0c57e3741b3ce8" gracePeriod=25 Nov 02 22:01:10.614804 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:01:10.614772 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:01:10.649313 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:10.649279 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" podUID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:01:10.649461 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:10.649291 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" podUID="dad61665-5a6c-4452-a0b5-32b8cd27308f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:01:12.926173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:12.926142 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:13.078000 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:13.077930 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:13.200138 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:13.200017 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:17.630284 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.630223 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:17.953942 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.953860 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl"] Nov 02 22:01:17.954578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954552 4441 cpu_manager.go:414] "containerMap: removing container" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" Nov 02 22:01:17.954578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954577 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" Nov 02 22:01:17.954756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954592 4441 cpu_manager.go:414] "containerMap: removing container" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" Nov 02 22:01:17.954756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954599 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" Nov 02 22:01:17.954756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954626 4441 cpu_manager.go:414] "containerMap: removing container" podUID="bebaa666-c24e-4436-9943-5c91c74b6f89" containerName="revoke-keys" Nov 02 22:01:17.954756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954633 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebaa666-c24e-4436-9943-5c91c74b6f89" containerName="revoke-keys" Nov 02 22:01:17.954995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954829 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" Nov 02 22:01:17.954995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954843 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="03da32a5-06a1-46e4-8379-154d9524ad4d" containerName="istio-proxy" Nov 02 22:01:17.954995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.954855 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebaa666-c24e-4436-9943-5c91c74b6f89" containerName="revoke-keys" Nov 02 22:01:17.959827 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:17.959805 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.012443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.012415 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl"] Nov 02 22:01:18.054491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.054454 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:01:18.054491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.054479 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:01:18.054712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.054675 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:01:18.056085 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.056068 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:01:18.059709 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.059688 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.059831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.059812 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.059908 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.059864 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4ws\" (UniqueName: \"kubernetes.io/projected/dce663d4-2c56-45b4-9252-10e149239cac-kube-api-access-dp4ws\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.161326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.161293 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4ws\" (UniqueName: \"kubernetes.io/projected/dce663d4-2c56-45b4-9252-10e149239cac-kube-api-access-dp4ws\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.161518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.161480 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.161614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.161595 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.162203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.162177 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.163900 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.163878 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.215496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.215435 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4ws\" (UniqueName: \"kubernetes.io/projected/dce663d4-2c56-45b4-9252-10e149239cac-kube-api-access-dp4ws\") pod \"maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.273387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.273357 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:18.643428 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:01:18.643403 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce663d4_2c56_45b4_9252_10e149239cac.slice/crio-102de739f3602f477479c7bafea7d386ba6de33abe702b9442d33d5190405f37 WatchSource:0}: Error finding container 102de739f3602f477479c7bafea7d386ba6de33abe702b9442d33d5190405f37: Status 404 returned error can't find the container with id 102de739f3602f477479c7bafea7d386ba6de33abe702b9442d33d5190405f37 Nov 02 22:01:18.644219 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:18.644195 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl"] Nov 02 22:01:19.102899 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:19.102860 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" event={"ID":"dce663d4-2c56-45b4-9252-10e149239cac","Type":"ContainerStarted","Data":"a47b46d34d17c8818edcdaf2257080fe5855d64bbcc5349879816b904071827d"} Nov 02 22:01:19.103110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:19.102916 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" event={"ID":"dce663d4-2c56-45b4-9252-10e149239cac","Type":"ContainerStarted","Data":"102de739f3602f477479c7bafea7d386ba6de33abe702b9442d33d5190405f37"} Nov 02 22:01:19.211621 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:19.211517 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" podStartSLOduration=2.211503577 podStartE2EDuration="2.211503577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:19.209715052 +0000 UTC m=+2908.765343389" watchObservedRunningTime="2026-11-02 22:01:19.211503577 +0000 UTC m=+2908.767131947" Nov 02 22:01:20.111630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:20.111592 4441 generic.go:356] "Generic (PLEG): container finished" podID="dce663d4-2c56-45b4-9252-10e149239cac" containerID="a47b46d34d17c8818edcdaf2257080fe5855d64bbcc5349879816b904071827d" exitCode=0 Nov 02 22:01:20.112042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:20.111681 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" event={"ID":"dce663d4-2c56-45b4-9252-10e149239cac","Type":"ContainerDied","Data":"a47b46d34d17c8818edcdaf2257080fe5855d64bbcc5349879816b904071827d"} Nov 02 22:01:20.648546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:20.648511 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" podUID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:01:20.648722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:20.648518 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" podUID="dad61665-5a6c-4452-a0b5-32b8cd27308f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:01:21.265660 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.265635 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:21.399837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.399753 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce663d4-2c56-45b4-9252-10e149239cac-kube-api-access-dp4ws\" (UniqueName: \"kubernetes.io/projected/dce663d4-2c56-45b4-9252-10e149239cac-kube-api-access-dp4ws\") pod \"dce663d4-2c56-45b4-9252-10e149239cac\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " Nov 02 22:01:21.399837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.399809 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-xks-ca\") pod \"dce663d4-2c56-45b4-9252-10e149239cac\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " Nov 02 22:01:21.400090 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.399871 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-openshift-ca\") pod \"dce663d4-2c56-45b4-9252-10e149239cac\" (UID: \"dce663d4-2c56-45b4-9252-10e149239cac\") " Nov 02 22:01:21.400493 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.400463 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-openshift-ca" pod "dce663d4-2c56-45b4-9252-10e149239cac" (UID: "dce663d4-2c56-45b4-9252-10e149239cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:21.402505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.402474 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-xks-ca" pod "dce663d4-2c56-45b4-9252-10e149239cac" (UID: "dce663d4-2c56-45b4-9252-10e149239cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:01:21.402603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.402512 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce663d4-2c56-45b4-9252-10e149239cac-kube-api-access-dp4ws" pod "dce663d4-2c56-45b4-9252-10e149239cac" (UID: "dce663d4-2c56-45b4-9252-10e149239cac"). InnerVolumeSpecName "kube-api-access-dp4ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.501440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.501400 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4ws\" (UniqueName: \"kubernetes.io/projected/dce663d4-2c56-45b4-9252-10e149239cac-kube-api-access-dp4ws\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.501440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.501436 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.501626 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:21.501451 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce663d4-2c56-45b4-9252-10e149239cac-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:22.127160 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:22.127121 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" Nov 02 22:01:22.127160 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:22.127147 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl" event={"ID":"dce663d4-2c56-45b4-9252-10e149239cac","Type":"ContainerDied","Data":"102de739f3602f477479c7bafea7d386ba6de33abe702b9442d33d5190405f37"} Nov 02 22:01:22.127421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:22.127204 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="102de739f3602f477479c7bafea7d386ba6de33abe702b9442d33d5190405f37" Nov 02 22:01:22.783330 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:22.783229 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:27.433084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:27.433040 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:29.189307 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.189273 4441 generic.go:356] "Generic (PLEG): container finished" podID="dad61665-5a6c-4452-a0b5-32b8cd27308f" containerID="434625c44f9a6326f22cf66ccbfd3325a28e3c191b1fbaec7d1109f4a0e56b59" exitCode=0 Nov 02 22:01:29.189731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.189410 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" event={"ID":"dad61665-5a6c-4452-a0b5-32b8cd27308f","Type":"ContainerDied","Data":"434625c44f9a6326f22cf66ccbfd3325a28e3c191b1fbaec7d1109f4a0e56b59"} Nov 02 22:01:29.191497 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.191465 4441 generic.go:356] "Generic (PLEG): container finished" podID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" containerID="ac11e13931dbf956c7181015351243dac43cd1d0431b6d257d0c57e3741b3ce8" exitCode=0 Nov 02 22:01:29.191619 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.191521 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" event={"ID":"213a55c1-c58b-43bc-b2e2-222ea25f9d78","Type":"ContainerDied","Data":"ac11e13931dbf956c7181015351243dac43cd1d0431b6d257d0c57e3741b3ce8"} Nov 02 22:01:29.238263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.238231 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 22:01:29.346703 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.346672 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 22:01:29.388789 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.388752 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad61665-5a6c-4452-a0b5-32b8cd27308f-kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/dad61665-5a6c-4452-a0b5-32b8cd27308f-kube-api-access-zv26d\") pod \"dad61665-5a6c-4452-a0b5-32b8cd27308f\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " Nov 02 22:01:29.389048 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.388867 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad61665-5a6c-4452-a0b5-32b8cd27308f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad61665-5a6c-4452-a0b5-32b8cd27308f-plugins-config-volume\") pod \"dad61665-5a6c-4452-a0b5-32b8cd27308f\" (UID: \"dad61665-5a6c-4452-a0b5-32b8cd27308f\") " Nov 02 22:01:29.389561 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.389529 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad61665-5a6c-4452-a0b5-32b8cd27308f-plugins-config-volume" pod "dad61665-5a6c-4452-a0b5-32b8cd27308f" (UID: "dad61665-5a6c-4452-a0b5-32b8cd27308f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:29.391575 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.391549 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad61665-5a6c-4452-a0b5-32b8cd27308f-kube-api-access-zv26d" pod "dad61665-5a6c-4452-a0b5-32b8cd27308f" (UID: "dad61665-5a6c-4452-a0b5-32b8cd27308f"). InnerVolumeSpecName "kube-api-access-zv26d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:29.490624 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.490544 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213a55c1-c58b-43bc-b2e2-222ea25f9d78-kube-api-access-kg2wx\" (UniqueName: \"kubernetes.io/projected/213a55c1-c58b-43bc-b2e2-222ea25f9d78-kube-api-access-kg2wx\") pod \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " Nov 02 22:01:29.490800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.490677 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213a55c1-c58b-43bc-b2e2-222ea25f9d78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213a55c1-c58b-43bc-b2e2-222ea25f9d78-plugins-config-volume\") pod \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\" (UID: \"213a55c1-c58b-43bc-b2e2-222ea25f9d78\") " Nov 02 22:01:29.491151 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.491101 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213a55c1-c58b-43bc-b2e2-222ea25f9d78-plugins-config-volume" pod "213a55c1-c58b-43bc-b2e2-222ea25f9d78" (UID: "213a55c1-c58b-43bc-b2e2-222ea25f9d78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:29.491151 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.491136 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/dad61665-5a6c-4452-a0b5-32b8cd27308f-kube-api-access-zv26d\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:29.491362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.491161 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad61665-5a6c-4452-a0b5-32b8cd27308f-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:29.493256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.493231 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213a55c1-c58b-43bc-b2e2-222ea25f9d78-kube-api-access-kg2wx" pod "213a55c1-c58b-43bc-b2e2-222ea25f9d78" (UID: "213a55c1-c58b-43bc-b2e2-222ea25f9d78"). InnerVolumeSpecName "kube-api-access-kg2wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:29.591800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.591764 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg2wx\" (UniqueName: \"kubernetes.io/projected/213a55c1-c58b-43bc-b2e2-222ea25f9d78-kube-api-access-kg2wx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:29.591800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.591794 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213a55c1-c58b-43bc-b2e2-222ea25f9d78-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:29.704062 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.704030 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:29.822443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:29.822408 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:30.199876 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.199788 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" Nov 02 22:01:30.199876 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.199832 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb" event={"ID":"213a55c1-c58b-43bc-b2e2-222ea25f9d78","Type":"ContainerDied","Data":"5774b1467e519f41459df20e561d02c39923a354f4304b2c4eaf86212e551f82"} Nov 02 22:01:30.200405 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.199898 4441 scope.go:122] "RemoveContainer" containerID="ac11e13931dbf956c7181015351243dac43cd1d0431b6d257d0c57e3741b3ce8" Nov 02 22:01:30.201664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.201633 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" event={"ID":"dad61665-5a6c-4452-a0b5-32b8cd27308f","Type":"ContainerDied","Data":"166c19fdc60b0caa60cc6018bfb8e5b382ab3aa085e6bd1611ba8800f409b86f"} Nov 02 22:01:30.201664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.201655 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg" Nov 02 22:01:30.212618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.212600 4441 scope.go:122] "RemoveContainer" containerID="434625c44f9a6326f22cf66ccbfd3325a28e3c191b1fbaec7d1109f4a0e56b59" Nov 02 22:01:30.357824 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.357798 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg"] Nov 02 22:01:30.363605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.363578 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4f799-56b5f49845-8vdrg"] Nov 02 22:01:30.467054 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.464210 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb"] Nov 02 22:01:30.469200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.469176 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4f799-54b658bd4c-hkmkb"] Nov 02 22:01:30.585810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.585781 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" path="/var/lib/kubelet/pods/213a55c1-c58b-43bc-b2e2-222ea25f9d78/volumes" Nov 02 22:01:30.586190 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:30.586175 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad61665-5a6c-4452-a0b5-32b8cd27308f" path="/var/lib/kubelet/pods/dad61665-5a6c-4452-a0b5-32b8cd27308f/volumes" Nov 02 22:01:31.895214 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:31.895170 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:34.630241 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.630207 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc"] Nov 02 22:01:34.630994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.630962 4441 cpu_manager.go:414] "containerMap: removing container" podUID="dce663d4-2c56-45b4-9252-10e149239cac" containerName="revoke-keys" Nov 02 22:01:34.631044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.630997 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce663d4-2c56-45b4-9252-10e149239cac" containerName="revoke-keys" Nov 02 22:01:34.631044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.631032 4441 cpu_manager.go:414] "containerMap: removing container" podUID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" containerName="payload-pre-processing" Nov 02 22:01:34.631123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.631040 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" containerName="payload-pre-processing" Nov 02 22:01:34.631123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.631055 4441 cpu_manager.go:414] "containerMap: removing container" podUID="dad61665-5a6c-4452-a0b5-32b8cd27308f" containerName="payload-processing" Nov 02 22:01:34.631123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.631062 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad61665-5a6c-4452-a0b5-32b8cd27308f" containerName="payload-processing" Nov 02 22:01:34.631221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.631182 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce663d4-2c56-45b4-9252-10e149239cac" containerName="revoke-keys" Nov 02 22:01:34.631221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.631200 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="213a55c1-c58b-43bc-b2e2-222ea25f9d78" containerName="payload-pre-processing" Nov 02 22:01:34.631221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.631211 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad61665-5a6c-4452-a0b5-32b8cd27308f" containerName="payload-processing" Nov 02 22:01:34.636827 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.636802 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.644009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.643983 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ns8l\" (UniqueName: \"kubernetes.io/projected/93cdc737-0552-4fde-942c-dd7b0f71a635-kube-api-access-8ns8l\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.644221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.644193 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.644335 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.644258 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.685172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.685145 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc"] Nov 02 22:01:34.727639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.727608 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:01:34.727759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.727670 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:01:34.727954 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.727940 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:01:34.728085 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.728067 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:01:34.745244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.745216 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.745447 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.745426 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.745710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.745681 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ns8l\" (UniqueName: \"kubernetes.io/projected/93cdc737-0552-4fde-942c-dd7b0f71a635-kube-api-access-8ns8l\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.745995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.745954 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.748187 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.748159 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.800564 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.800535 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ns8l\" (UniqueName: \"kubernetes.io/projected/93cdc737-0552-4fde-942c-dd7b0f71a635-kube-api-access-8ns8l\") pod \"maas-api-revoke-sub-ecd8b1d3-mwfkc\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:34.950851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:34.950788 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:35.338677 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:01:35.338643 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cdc737_0552_4fde_942c_dd7b0f71a635.slice/crio-07040c84d6dfdcf9b2dedcddc83b81008edd74b8d6cf1143173b2ebc8cba2810 WatchSource:0}: Error finding container 07040c84d6dfdcf9b2dedcddc83b81008edd74b8d6cf1143173b2ebc8cba2810: Status 404 returned error can't find the container with id 07040c84d6dfdcf9b2dedcddc83b81008edd74b8d6cf1143173b2ebc8cba2810 Nov 02 22:01:35.339589 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:35.339560 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc"] Nov 02 22:01:36.030770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.030679 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4"] Nov 02 22:01:36.033763 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.033746 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.061381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.061358 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.061503 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.061484 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.061562 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.061550 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzhf\" (UniqueName: \"kubernetes.io/projected/5f9b40c8-bb5d-44f7-824b-273d24bea51f-kube-api-access-gbzhf\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.101113 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.100319 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4"] Nov 02 22:01:36.162173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.162138 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.162341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.162230 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzhf\" (UniqueName: \"kubernetes.io/projected/5f9b40c8-bb5d-44f7-824b-273d24bea51f-kube-api-access-gbzhf\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.162423 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.162398 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.162881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.162862 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.164742 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.164723 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.216259 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.216224 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzhf\" (UniqueName: \"kubernetes.io/projected/5f9b40c8-bb5d-44f7-824b-273d24bea51f-kube-api-access-gbzhf\") pod \"maas-api-revoke-sub-a27f2f5c-nrkq4\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.256198 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.256159 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" event={"ID":"93cdc737-0552-4fde-942c-dd7b0f71a635","Type":"ContainerStarted","Data":"1d2fb41f71d9ea35e272300c4cbd390923d21d04c539bd590ce10220d7969ec5"} Nov 02 22:01:36.256354 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.256221 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" event={"ID":"93cdc737-0552-4fde-942c-dd7b0f71a635","Type":"ContainerStarted","Data":"07040c84d6dfdcf9b2dedcddc83b81008edd74b8d6cf1143173b2ebc8cba2810"} Nov 02 22:01:36.346661 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.346635 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:36.365953 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.365909 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" podStartSLOduration=2.3658951950000002 podStartE2EDuration="2.365895195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:36.362950174 +0000 UTC m=+2925.918578511" watchObservedRunningTime="2026-11-02 22:01:36.365895195 +0000 UTC m=+2925.921523531" Nov 02 22:01:36.402230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.400982 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:36.715464 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:01:36.715432 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9b40c8_bb5d_44f7_824b_273d24bea51f.slice/crio-b7b115b84bb3780c8bf9d058df3b996645329a9e01076ecdcaaa0bc87c813721 WatchSource:0}: Error finding container b7b115b84bb3780c8bf9d058df3b996645329a9e01076ecdcaaa0bc87c813721: Status 404 returned error can't find the container with id b7b115b84bb3780c8bf9d058df3b996645329a9e01076ecdcaaa0bc87c813721 Nov 02 22:01:36.718258 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:36.718174 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4"] Nov 02 22:01:37.265065 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:37.265029 4441 generic.go:356] "Generic (PLEG): container finished" podID="93cdc737-0552-4fde-942c-dd7b0f71a635" containerID="1d2fb41f71d9ea35e272300c4cbd390923d21d04c539bd590ce10220d7969ec5" exitCode=0 Nov 02 22:01:37.265506 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:37.265115 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" event={"ID":"93cdc737-0552-4fde-942c-dd7b0f71a635","Type":"ContainerDied","Data":"1d2fb41f71d9ea35e272300c4cbd390923d21d04c539bd590ce10220d7969ec5"} Nov 02 22:01:37.266796 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:37.266765 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" event={"ID":"5f9b40c8-bb5d-44f7-824b-273d24bea51f","Type":"ContainerStarted","Data":"0af6bc83c61080003038ca2880e8d79660138c40637353231ad23376c25ce32c"} Nov 02 22:01:37.266939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:37.266815 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" event={"ID":"5f9b40c8-bb5d-44f7-824b-273d24bea51f","Type":"ContainerStarted","Data":"b7b115b84bb3780c8bf9d058df3b996645329a9e01076ecdcaaa0bc87c813721"} Nov 02 22:01:37.482084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:37.481960 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" podStartSLOduration=2.48194185 podStartE2EDuration="2.48194185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:37.48005674 +0000 UTC m=+2927.035685078" watchObservedRunningTime="2026-11-02 22:01:37.48194185 +0000 UTC m=+2927.037570187" Nov 02 22:01:38.276637 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.276597 4441 generic.go:356] "Generic (PLEG): container finished" podID="5f9b40c8-bb5d-44f7-824b-273d24bea51f" containerID="0af6bc83c61080003038ca2880e8d79660138c40637353231ad23376c25ce32c" exitCode=0 Nov 02 22:01:38.277062 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.276677 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" event={"ID":"5f9b40c8-bb5d-44f7-824b-273d24bea51f","Type":"ContainerDied","Data":"0af6bc83c61080003038ca2880e8d79660138c40637353231ad23376c25ce32c"} Nov 02 22:01:38.444173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.444144 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:38.489845 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.489808 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-openshift-ca\") pod \"93cdc737-0552-4fde-942c-dd7b0f71a635\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " Nov 02 22:01:38.490041 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.489927 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-xks-ca\") pod \"93cdc737-0552-4fde-942c-dd7b0f71a635\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " Nov 02 22:01:38.490148 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.490128 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cdc737-0552-4fde-942c-dd7b0f71a635-kube-api-access-8ns8l\" (UniqueName: \"kubernetes.io/projected/93cdc737-0552-4fde-942c-dd7b0f71a635-kube-api-access-8ns8l\") pod \"93cdc737-0552-4fde-942c-dd7b0f71a635\" (UID: \"93cdc737-0552-4fde-942c-dd7b0f71a635\") " Nov 02 22:01:38.490275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.490133 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-openshift-ca" pod "93cdc737-0552-4fde-942c-dd7b0f71a635" (UID: "93cdc737-0552-4fde-942c-dd7b0f71a635"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:38.490638 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.490613 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:38.491932 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.491909 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-xks-ca" pod "93cdc737-0552-4fde-942c-dd7b0f71a635" (UID: "93cdc737-0552-4fde-942c-dd7b0f71a635"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:01:38.492058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.491996 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cdc737-0552-4fde-942c-dd7b0f71a635-kube-api-access-8ns8l" pod "93cdc737-0552-4fde-942c-dd7b0f71a635" (UID: "93cdc737-0552-4fde-942c-dd7b0f71a635"). InnerVolumeSpecName "kube-api-access-8ns8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:38.591910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.591876 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ns8l\" (UniqueName: \"kubernetes.io/projected/93cdc737-0552-4fde-942c-dd7b0f71a635-kube-api-access-8ns8l\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:38.591910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:38.591910 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cdc737-0552-4fde-942c-dd7b0f71a635-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:39.287058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.287021 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" Nov 02 22:01:39.287483 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.287029 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc" event={"ID":"93cdc737-0552-4fde-942c-dd7b0f71a635","Type":"ContainerDied","Data":"07040c84d6dfdcf9b2dedcddc83b81008edd74b8d6cf1143173b2ebc8cba2810"} Nov 02 22:01:39.287483 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.287155 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07040c84d6dfdcf9b2dedcddc83b81008edd74b8d6cf1143173b2ebc8cba2810" Nov 02 22:01:39.449726 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.449702 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:39.503333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.503307 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-openshift-ca\") pod \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " Nov 02 22:01:39.503495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.503407 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9b40c8-bb5d-44f7-824b-273d24bea51f-kube-api-access-gbzhf\" (UniqueName: \"kubernetes.io/projected/5f9b40c8-bb5d-44f7-824b-273d24bea51f-kube-api-access-gbzhf\") pod \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " Nov 02 22:01:39.503563 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.503490 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-xks-ca\") pod \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\" (UID: \"5f9b40c8-bb5d-44f7-824b-273d24bea51f\") " Nov 02 22:01:39.503739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.503708 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-openshift-ca" pod "5f9b40c8-bb5d-44f7-824b-273d24bea51f" (UID: "5f9b40c8-bb5d-44f7-824b-273d24bea51f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:39.504055 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.504031 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:39.505677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.505640 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9b40c8-bb5d-44f7-824b-273d24bea51f-kube-api-access-gbzhf" pod "5f9b40c8-bb5d-44f7-824b-273d24bea51f" (UID: "5f9b40c8-bb5d-44f7-824b-273d24bea51f"). InnerVolumeSpecName "kube-api-access-gbzhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:39.505786 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.505696 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-xks-ca" pod "5f9b40c8-bb5d-44f7-824b-273d24bea51f" (UID: "5f9b40c8-bb5d-44f7-824b-273d24bea51f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:01:39.604902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.604869 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzhf\" (UniqueName: \"kubernetes.io/projected/5f9b40c8-bb5d-44f7-824b-273d24bea51f-kube-api-access-gbzhf\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:39.604902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:39.604899 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9b40c8-bb5d-44f7-824b-273d24bea51f-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:40.299541 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:40.299508 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" Nov 02 22:01:40.299541 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:40.299531 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4" event={"ID":"5f9b40c8-bb5d-44f7-824b-273d24bea51f","Type":"ContainerDied","Data":"b7b115b84bb3780c8bf9d058df3b996645329a9e01076ecdcaaa0bc87c813721"} Nov 02 22:01:40.299998 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:40.299581 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7b115b84bb3780c8bf9d058df3b996645329a9e01076ecdcaaa0bc87c813721" Nov 02 22:01:40.619081 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:01:40.619038 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:01:40.904723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:40.904640 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:41.839319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:41.839288 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:41.839932 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:41.839910 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4"] Nov 02 22:01:44.722496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:44.722412 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:44.840196 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:44.840160 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:45.395219 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:45.395178 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:49.883493 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:49.883457 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:52.390759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:52.390725 4441 generic.go:356] "Generic (PLEG): container finished" podID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerID="f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc" exitCode=0 Nov 02 22:01:52.391225 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:52.390805 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" event={"ID":"2aac1cb0-9792-4d93-b66e-259a1d6a6500","Type":"ContainerDied","Data":"f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc"} Nov 02 22:01:52.391315 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:52.391299 4441 scope.go:122] "RemoveContainer" containerID="f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc" Nov 02 22:01:52.392979 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:52.392936 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:01:52.393125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:52.393020 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:01:52.393125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:52.393056 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:01:53.406420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:53.406383 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" event={"ID":"2aac1cb0-9792-4d93-b66e-259a1d6a6500","Type":"ContainerStarted","Data":"9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68"} Nov 02 22:01:53.933759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:53.933726 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 22:01:53.934038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:53.934009 4441 patch_prober.go:29] interesting pod/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 02 22:01:53.934153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:53.934072 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 02 22:01:54.395514 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:54.395481 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:54.933223 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:54.933189 4441 patch_prober.go:29] interesting pod/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 02 22:01:54.933617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:54.933246 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 02 22:01:55.933433 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:55.933399 4441 patch_prober.go:29] interesting pod/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 02 22:01:55.933890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:55.933469 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 02 22:01:56.859766 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:56.859705 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" podUID="e7e53ada-cf10-4902-9c6d-6bb378fc87ce" containerName="main" containerID="cri-o://e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6" gracePeriod=45 Nov 02 22:01:56.933439 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:56.933409 4441 patch_prober.go:29] interesting pod/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 02 22:01:56.933862 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:56.933463 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 02 22:01:57.043126 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.043076 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:57.230052 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.230026 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 22:01:57.255772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.255750 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tls-certs\") pod \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " Nov 02 22:01:57.255897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.255816 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tmp-dir\") pod \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " Nov 02 22:01:57.255897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.255857 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-kube-api-access-dfhjn\" (UniqueName: \"kubernetes.io/projected/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-kube-api-access-dfhjn\") pod \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " Nov 02 22:01:57.255897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.255881 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-home\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-home\") pod \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " Nov 02 22:01:57.256084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.255957 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-dshm\") pod \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " Nov 02 22:01:57.256084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.256027 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-model-cache\") pod \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\" (UID: \"e7e53ada-cf10-4902-9c6d-6bb378fc87ce\") " Nov 02 22:01:57.256238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.256206 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tmp-dir" pod "e7e53ada-cf10-4902-9c6d-6bb378fc87ce" (UID: "e7e53ada-cf10-4902-9c6d-6bb378fc87ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:57.256380 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.256206 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-home" pod "e7e53ada-cf10-4902-9c6d-6bb378fc87ce" (UID: "e7e53ada-cf10-4902-9c6d-6bb378fc87ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:57.256452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.256392 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-model-cache" pod "e7e53ada-cf10-4902-9c6d-6bb378fc87ce" (UID: "e7e53ada-cf10-4902-9c6d-6bb378fc87ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:57.256515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.256487 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:57.256515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.256502 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:57.258226 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.258204 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-dshm" pod "e7e53ada-cf10-4902-9c6d-6bb378fc87ce" (UID: "e7e53ada-cf10-4902-9c6d-6bb378fc87ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:57.258316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.258203 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tls-certs" pod "e7e53ada-cf10-4902-9c6d-6bb378fc87ce" (UID: "e7e53ada-cf10-4902-9c6d-6bb378fc87ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:01:57.258316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.258257 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-kube-api-access-dfhjn" pod "e7e53ada-cf10-4902-9c6d-6bb378fc87ce" (UID: "e7e53ada-cf10-4902-9c6d-6bb378fc87ce"). InnerVolumeSpecName "kube-api-access-dfhjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:57.357956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.357925 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:57.357956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.357954 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:57.357956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.357978 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:57.358203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.357988 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfhjn\" (UniqueName: \"kubernetes.io/projected/e7e53ada-cf10-4902-9c6d-6bb378fc87ce-kube-api-access-dfhjn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:57.437527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.437485 4441 generic.go:356] "Generic (PLEG): container finished" podID="e7e53ada-cf10-4902-9c6d-6bb378fc87ce" containerID="e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6" exitCode=0 Nov 02 22:01:57.437702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.437532 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" Nov 02 22:01:57.437702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.437567 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" event={"ID":"e7e53ada-cf10-4902-9c6d-6bb378fc87ce","Type":"ContainerDied","Data":"e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6"} Nov 02 22:01:57.437702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.437612 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4" event={"ID":"e7e53ada-cf10-4902-9c6d-6bb378fc87ce","Type":"ContainerDied","Data":"2cd40af9cacc75bcd4d33ea8b26c925f82c4313d250e1b869c4294bc36201d6f"} Nov 02 22:01:57.437702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.437629 4441 scope.go:122] "RemoveContainer" containerID="e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6" Nov 02 22:01:57.448526 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.448506 4441 scope.go:122] "RemoveContainer" containerID="e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6" Nov 02 22:01:57.448806 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:01:57.448788 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6\": container with ID starting with e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6 not found: ID does not exist" containerID="e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6" Nov 02 22:01:57.448898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.448815 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6"} err="failed to get container status \"e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6\": rpc error: code = NotFound desc = could not find container \"e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6\": container with ID starting with e147057a6c8acc2154ce61de5aaafc5243a94a546666eedc80f32d85d7f97fe6 not found: ID does not exist" Nov 02 22:01:57.602326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.602296 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4"] Nov 02 22:01:57.616502 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.606645 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-11537e/test-model-11537e-kserve-566cb5694c-btql4"] Nov 02 22:01:57.933109 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.933033 4441 patch_prober.go:29] interesting pod/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 02 22:01:57.933109 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:57.933083 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 02 22:01:58.586629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:58.586592 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e53ada-cf10-4902-9c6d-6bb378fc87ce" path="/var/lib/kubelet/pods/e7e53ada-cf10-4902-9c6d-6bb378fc87ce/volumes" Nov 02 22:01:58.852480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:58.852386 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:58.938224 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:58.938189 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 22:01:58.938574 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:58.938553 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 22:01:58.939304 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:01:58.939286 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 22:02:02.654529 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:02.654489 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:02.793516 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:02.793477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:04.276670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:04.276637 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:04.530532 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:04.530462 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:04.658504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:04.658473 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:04.764504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:04.764463 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8"] Nov 02 22:02:04.811303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:04.811272 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4f799-298j8"] Nov 02 22:02:05.138693 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:05.138612 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:06.583704 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:06.583670 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebaa666-c24e-4436-9943-5c91c74b6f89" path="/var/lib/kubelet/pods/bebaa666-c24e-4436-9943-5c91c74b6f89/volumes" Nov 02 22:02:08.591125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:08.591093 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:10.075115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:10.075083 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:10.312983 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:10.312925 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:10.442336 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:10.442233 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:10.617148 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:10.617109 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:02:10.617891 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:10.617859 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl"] Nov 02 22:02:10.664515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:10.664484 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-686bbf-hczjl"] Nov 02 22:02:10.904431 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:10.904373 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.230701 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:11.230599 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.353955 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:11.353920 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.653627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:11.653596 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.654145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:11.654125 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn"] Nov 02 22:02:11.654349 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:11.654332 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" containerID="cri-o://ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe" gracePeriod=30 Nov 02 22:02:12.356074 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356042 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9"] Nov 02 22:02:12.356637 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356620 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e7e53ada-cf10-4902-9c6d-6bb378fc87ce" containerName="main" Nov 02 22:02:12.356691 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356638 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e53ada-cf10-4902-9c6d-6bb378fc87ce" containerName="main" Nov 02 22:02:12.356691 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356662 4441 cpu_manager.go:414] "containerMap: removing container" podUID="5f9b40c8-bb5d-44f7-824b-273d24bea51f" containerName="revoke-keys" Nov 02 22:02:12.356691 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356672 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9b40c8-bb5d-44f7-824b-273d24bea51f" containerName="revoke-keys" Nov 02 22:02:12.356691 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356686 4441 cpu_manager.go:414] "containerMap: removing container" podUID="93cdc737-0552-4fde-942c-dd7b0f71a635" containerName="revoke-keys" Nov 02 22:02:12.356817 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356691 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cdc737-0552-4fde-942c-dd7b0f71a635" containerName="revoke-keys" Nov 02 22:02:12.356817 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356806 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e53ada-cf10-4902-9c6d-6bb378fc87ce" containerName="main" Nov 02 22:02:12.356890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356821 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9b40c8-bb5d-44f7-824b-273d24bea51f" containerName="revoke-keys" Nov 02 22:02:12.356890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.356831 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cdc737-0552-4fde-942c-dd7b0f71a635" containerName="revoke-keys" Nov 02 22:02:12.360457 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.360436 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.410153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.410129 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:02:12.410427 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.410295 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:02:12.410609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.410389 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:02:12.410609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.410413 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:02:12.413557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.413537 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9"] Nov 02 22:02:12.467737 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.467683 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.467857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.467753 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkq2\" (UniqueName: \"kubernetes.io/projected/a6f9d8e0-40e5-4515-b2ce-ac258624223f-kube-api-access-zvkq2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.467857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.467794 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.568587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.568548 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.568731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.568599 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkq2\" (UniqueName: \"kubernetes.io/projected/a6f9d8e0-40e5-4515-b2ce-ac258624223f-kube-api-access-zvkq2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.568731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.568646 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.569334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.569310 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.570798 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.570780 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.593698 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.593664 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce663d4-2c56-45b4-9252-10e149239cac" path="/var/lib/kubelet/pods/dce663d4-2c56-45b4-9252-10e149239cac/volumes" Nov 02 22:02:12.621776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.621724 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkq2\" (UniqueName: \"kubernetes.io/projected/a6f9d8e0-40e5-4515-b2ce-ac258624223f-kube-api-access-zvkq2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:12.672628 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:12.672602 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:13.039356 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:13.039321 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9"] Nov 02 22:02:13.039535 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:13.039364 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f9d8e0_40e5_4515_b2ce_ac258624223f.slice/crio-ae84f505b648c6bba1275b641d5ec2970d533a822e0c34bd5b80327425749400 WatchSource:0}: Error finding container ae84f505b648c6bba1275b641d5ec2970d533a822e0c34bd5b80327425749400: Status 404 returned error can't find the container with id ae84f505b648c6bba1275b641d5ec2970d533a822e0c34bd5b80327425749400 Nov 02 22:02:13.606600 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:13.606568 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" event={"ID":"a6f9d8e0-40e5-4515-b2ce-ac258624223f","Type":"ContainerStarted","Data":"d702d0a30f102fba7056042b2386f3bd8b0ec9d23228428fd71b2938cf89f5f8"} Nov 02 22:02:13.607015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:13.606622 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" event={"ID":"a6f9d8e0-40e5-4515-b2ce-ac258624223f","Type":"ContainerStarted","Data":"ae84f505b648c6bba1275b641d5ec2970d533a822e0c34bd5b80327425749400"} Nov 02 22:02:13.714347 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:13.714258 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" podStartSLOduration=1.7142463829999999 podStartE2EDuration="1.714246383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:13.712747495 +0000 UTC m=+2963.268375832" watchObservedRunningTime="2026-11-02 22:02:13.714246383 +0000 UTC m=+2963.269874719" Nov 02 22:02:14.615153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:14.615112 4441 generic.go:356] "Generic (PLEG): container finished" podID="a6f9d8e0-40e5-4515-b2ce-ac258624223f" containerID="d702d0a30f102fba7056042b2386f3bd8b0ec9d23228428fd71b2938cf89f5f8" exitCode=0 Nov 02 22:02:14.615547 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:14.615240 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" event={"ID":"a6f9d8e0-40e5-4515-b2ce-ac258624223f","Type":"ContainerDied","Data":"d702d0a30f102fba7056042b2386f3bd8b0ec9d23228428fd71b2938cf89f5f8"} Nov 02 22:02:15.386712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.386677 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:15.766248 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.766226 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:15.803183 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.803156 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-openshift-ca\") pod \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " Nov 02 22:02:15.803302 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.803246 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f9d8e0-40e5-4515-b2ce-ac258624223f-kube-api-access-zvkq2\" (UniqueName: \"kubernetes.io/projected/a6f9d8e0-40e5-4515-b2ce-ac258624223f-kube-api-access-zvkq2\") pod \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " Nov 02 22:02:15.803302 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.803297 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-xks-ca\") pod \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\" (UID: \"a6f9d8e0-40e5-4515-b2ce-ac258624223f\") " Nov 02 22:02:15.803490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.803468 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-openshift-ca" pod "a6f9d8e0-40e5-4515-b2ce-ac258624223f" (UID: "a6f9d8e0-40e5-4515-b2ce-ac258624223f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:15.803627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.803607 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:15.805301 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.805282 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-xks-ca" pod "a6f9d8e0-40e5-4515-b2ce-ac258624223f" (UID: "a6f9d8e0-40e5-4515-b2ce-ac258624223f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:15.805384 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.805307 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f9d8e0-40e5-4515-b2ce-ac258624223f-kube-api-access-zvkq2" pod "a6f9d8e0-40e5-4515-b2ce-ac258624223f" (UID: "a6f9d8e0-40e5-4515-b2ce-ac258624223f"). InnerVolumeSpecName "kube-api-access-zvkq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:15.904961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.904931 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f9d8e0-40e5-4515-b2ce-ac258624223f-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:15.905104 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:15.904992 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvkq2\" (UniqueName: \"kubernetes.io/projected/a6f9d8e0-40e5-4515-b2ce-ac258624223f-kube-api-access-zvkq2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:16.518400 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.518368 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5"] Nov 02 22:02:16.519104 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.519083 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a6f9d8e0-40e5-4515-b2ce-ac258624223f" containerName="revoke-keys" Nov 02 22:02:16.519202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.519104 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f9d8e0-40e5-4515-b2ce-ac258624223f" containerName="revoke-keys" Nov 02 22:02:16.519202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.519198 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f9d8e0-40e5-4515-b2ce-ac258624223f" containerName="revoke-keys" Nov 02 22:02:16.523026 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.523006 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.587647 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.587613 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5"] Nov 02 22:02:16.613334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.613302 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.613515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.613351 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2lr\" (UniqueName: \"kubernetes.io/projected/47d15082-8d63-409b-bc2c-fc2159d7cffa-kube-api-access-5p2lr\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.613515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.613417 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.640010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.639986 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" Nov 02 22:02:16.640162 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.639979 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9" event={"ID":"a6f9d8e0-40e5-4515-b2ce-ac258624223f","Type":"ContainerDied","Data":"ae84f505b648c6bba1275b641d5ec2970d533a822e0c34bd5b80327425749400"} Nov 02 22:02:16.640162 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.640121 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae84f505b648c6bba1275b641d5ec2970d533a822e0c34bd5b80327425749400" Nov 02 22:02:16.714089 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.714059 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.714222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.714120 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2lr\" (UniqueName: \"kubernetes.io/projected/47d15082-8d63-409b-bc2c-fc2159d7cffa-kube-api-access-5p2lr\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.714222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.714176 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.714895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.714863 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.716578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.716544 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.769105 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.769053 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2lr\" (UniqueName: \"kubernetes.io/projected/47d15082-8d63-409b-bc2c-fc2159d7cffa-kube-api-access-5p2lr\") pod \"maas-api-revoke-sub-32fac650-xn4z5\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.839386 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.839353 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:16.944073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:16.944050 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 22:02:17.017537 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017502 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-kube-api-access-fj5tt\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-kube-api-access-fj5tt\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.017690 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017596 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-certs\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.017690 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017639 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-data\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.017767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017744 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2719ffe-8af4-4118-bf17-be831cfb32da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2719ffe-8af4-4118-bf17-be831cfb32da-istiod-ca-cert\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.017804 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017784 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-socket\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.017866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017818 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-podinfo\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.017926 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017901 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-credential-socket\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.017926 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017885 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-certs" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:17.018091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017934 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-token\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-token\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.018091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.017963 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-data" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:17.018091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018045 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-envoy\") pod \"f2719ffe-8af4-4118-bf17-be831cfb32da\" (UID: \"f2719ffe-8af4-4118-bf17-be831cfb32da\") " Nov 02 22:02:17.018263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018197 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2719ffe-8af4-4118-bf17-be831cfb32da-istiod-ca-cert" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:17.018263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018219 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-credential-socket" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:17.018436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018393 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-socket" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:17.018530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018481 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2719ffe-8af4-4118-bf17-be831cfb32da-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.018530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018497 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.018530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018510 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.018530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018519 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.018530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.018527 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.020224 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.020178 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-token" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:17.020440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.020410 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-envoy" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:17.020545 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.020435 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-kube-api-access-fj5tt" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "kube-api-access-fj5tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:17.020545 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.020416 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-podinfo" pod "f2719ffe-8af4-4118-bf17-be831cfb32da" (UID: "f2719ffe-8af4-4118-bf17-be831cfb32da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:17.044601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.044557 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:17.119958 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.119923 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.119958 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.119954 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.120185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.119981 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj5tt\" (UniqueName: \"kubernetes.io/projected/f2719ffe-8af4-4118-bf17-be831cfb32da-kube-api-access-fj5tt\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.120185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.120009 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2719ffe-8af4-4118-bf17-be831cfb32da-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:17.181467 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.181422 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:17.225011 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:17.224964 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d15082_8d63_409b_bc2c_fc2159d7cffa.slice/crio-a926376805939d05ee5addc425036cb38a0375c6f923f792ae05387beecf5c4c WatchSource:0}: Error finding container a926376805939d05ee5addc425036cb38a0375c6f923f792ae05387beecf5c4c: Status 404 returned error can't find the container with id a926376805939d05ee5addc425036cb38a0375c6f923f792ae05387beecf5c4c Nov 02 22:02:17.227221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.227196 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5"] Nov 02 22:02:17.385483 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.385286 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:17.650553 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.650456 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" event={"ID":"47d15082-8d63-409b-bc2c-fc2159d7cffa","Type":"ContainerStarted","Data":"3f821c66aa5a356fde7ff16d986f91eff4e42d63e5b432fea8ef3c2c9ce38e07"} Nov 02 22:02:17.650553 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.650528 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" event={"ID":"47d15082-8d63-409b-bc2c-fc2159d7cffa","Type":"ContainerStarted","Data":"a926376805939d05ee5addc425036cb38a0375c6f923f792ae05387beecf5c4c"} Nov 02 22:02:17.652507 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.652480 4441 generic.go:356] "Generic (PLEG): container finished" podID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerID="ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe" exitCode=0 Nov 02 22:02:17.652665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.652539 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" Nov 02 22:02:17.652665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.652609 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" event={"ID":"f2719ffe-8af4-4118-bf17-be831cfb32da","Type":"ContainerDied","Data":"ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe"} Nov 02 22:02:17.652665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.652658 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn" event={"ID":"f2719ffe-8af4-4118-bf17-be831cfb32da","Type":"ContainerDied","Data":"ebf3fc4fe82ec486791a7edc6ed57eb2f8473da67dcf7d3fc2e99d5da6f3564c"} Nov 02 22:02:17.652795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.652688 4441 scope.go:122] "RemoveContainer" containerID="ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe" Nov 02 22:02:17.663598 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.663578 4441 scope.go:122] "RemoveContainer" containerID="7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16" Nov 02 22:02:17.673051 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.673024 4441 scope.go:122] "RemoveContainer" containerID="ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe" Nov 02 22:02:17.673314 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:17.673289 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe\": container with ID starting with ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe not found: ID does not exist" containerID="ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe" Nov 02 22:02:17.673416 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.673321 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe"} err="failed to get container status \"ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe\": rpc error: code = NotFound desc = could not find container \"ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe\": container with ID starting with ba58a473fef6047ba63bf110334f78bb2a0f1fb49bb620df0359dc09db00cefe not found: ID does not exist" Nov 02 22:02:17.673416 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.673337 4441 scope.go:122] "RemoveContainer" containerID="7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16" Nov 02 22:02:17.673626 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:17.673585 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16\": container with ID starting with 7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16 not found: ID does not exist" containerID="7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16" Nov 02 22:02:17.673684 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.673636 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16"} err="failed to get container status \"7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16\": rpc error: code = NotFound desc = could not find container \"7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16\": container with ID starting with 7ff14f46d4161998f91d324c3d93382947740f04dfb3fa1fe3883b4d83aa2e16 not found: ID does not exist" Nov 02 22:02:17.760854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.760798 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" podStartSLOduration=1.7607853100000002 podStartE2EDuration="1.76078531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:17.756964077 +0000 UTC m=+2967.312592413" watchObservedRunningTime="2026-11-02 22:02:17.76078531 +0000 UTC m=+2967.316413648" Nov 02 22:02:17.865393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.865359 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn"] Nov 02 22:02:17.872328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:17.872299 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d4f799-openshift-default-76569fd959-phrdn"] Nov 02 22:02:18.116038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.115994 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4"] Nov 02 22:02:18.116599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.116582 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" Nov 02 22:02:18.116599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.116599 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" Nov 02 22:02:18.116759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.116619 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" Nov 02 22:02:18.116759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.116626 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" Nov 02 22:02:18.116759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.116728 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" Nov 02 22:02:18.116759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.116739 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" containerName="istio-proxy" Nov 02 22:02:18.121649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.121627 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.145735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.145711 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m"] Nov 02 22:02:18.149570 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.149551 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.172780 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.172759 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4"] Nov 02 22:02:18.205002 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.204937 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m"] Nov 02 22:02:18.232473 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.232441 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.232637 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.232515 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.232725 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.232657 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.232725 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.232699 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.232992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.232781 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzl8m\" (UniqueName: \"kubernetes.io/projected/a976d014-017a-4cf3-bbf0-eec2fce0da38-kube-api-access-tzl8m\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.232992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.232808 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbj9z\" (UniqueName: \"kubernetes.io/projected/e5f37c3c-f91e-43c8-a8f2-33335d721702-kube-api-access-wbj9z\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.333805 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.333777 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.333962 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.333858 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.334138 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.334082 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.334207 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.334140 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.334207 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.334181 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzl8m\" (UniqueName: \"kubernetes.io/projected/a976d014-017a-4cf3-bbf0-eec2fce0da38-kube-api-access-tzl8m\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.334521 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.334213 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbj9z\" (UniqueName: \"kubernetes.io/projected/e5f37c3c-f91e-43c8-a8f2-33335d721702-kube-api-access-wbj9z\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.334745 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.334720 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.334956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.334934 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.337244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.337216 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.337356 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.337262 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.388510 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.388443 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbj9z\" (UniqueName: \"kubernetes.io/projected/e5f37c3c-f91e-43c8-a8f2-33335d721702-kube-api-access-wbj9z\") pod \"maas-api-revoke-sub-7b3f1ed1-t247m\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.388510 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.388443 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzl8m\" (UniqueName: \"kubernetes.io/projected/a976d014-017a-4cf3-bbf0-eec2fce0da38-kube-api-access-tzl8m\") pod \"maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.435277 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.435244 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:18.521579 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.521550 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:18.590450 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.590384 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2719ffe-8af4-4118-bf17-be831cfb32da" path="/var/lib/kubelet/pods/f2719ffe-8af4-4118-bf17-be831cfb32da/volumes" Nov 02 22:02:18.661677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.661587 4441 generic.go:356] "Generic (PLEG): container finished" podID="47d15082-8d63-409b-bc2c-fc2159d7cffa" containerID="3f821c66aa5a356fde7ff16d986f91eff4e42d63e5b432fea8ef3c2c9ce38e07" exitCode=0 Nov 02 22:02:18.661677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.661644 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" event={"ID":"47d15082-8d63-409b-bc2c-fc2159d7cffa","Type":"ContainerDied","Data":"3f821c66aa5a356fde7ff16d986f91eff4e42d63e5b432fea8ef3c2c9ce38e07"} Nov 02 22:02:18.804754 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.804722 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4"] Nov 02 22:02:18.805011 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:18.804988 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda976d014_017a_4cf3_bbf0_eec2fce0da38.slice/crio-fe952605a0f3609f31b1a97c376863e0365a0dcd0dc6f1f8802dc2b089ffec3b WatchSource:0}: Error finding container fe952605a0f3609f31b1a97c376863e0365a0dcd0dc6f1f8802dc2b089ffec3b: Status 404 returned error can't find the container with id fe952605a0f3609f31b1a97c376863e0365a0dcd0dc6f1f8802dc2b089ffec3b Nov 02 22:02:18.901681 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:18.901645 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f37c3c_f91e_43c8_a8f2_33335d721702.slice/crio-ad063b9c08ee440f59ebfff946f65109d6e4d097a589351b06f4a373be10d7c8 WatchSource:0}: Error finding container ad063b9c08ee440f59ebfff946f65109d6e4d097a589351b06f4a373be10d7c8: Status 404 returned error can't find the container with id ad063b9c08ee440f59ebfff946f65109d6e4d097a589351b06f4a373be10d7c8 Nov 02 22:02:18.903111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:18.903086 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m"] Nov 02 22:02:19.673642 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.673604 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" event={"ID":"e5f37c3c-f91e-43c8-a8f2-33335d721702","Type":"ContainerStarted","Data":"7e1cad03fe0e7449226abb1a499a3cad43da78ebb8d5a7099f59c46c3a2ef9d1"} Nov 02 22:02:19.673848 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.673676 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" event={"ID":"e5f37c3c-f91e-43c8-a8f2-33335d721702","Type":"ContainerStarted","Data":"ad063b9c08ee440f59ebfff946f65109d6e4d097a589351b06f4a373be10d7c8"} Nov 02 22:02:19.675517 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.675486 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" event={"ID":"a976d014-017a-4cf3-bbf0-eec2fce0da38","Type":"ContainerStarted","Data":"7759f2e24ba9d2a33404ddf1f094157f77b89ce62279a4ed1fa6f350f4f25719"} Nov 02 22:02:19.675677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.675540 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" event={"ID":"a976d014-017a-4cf3-bbf0-eec2fce0da38","Type":"ContainerStarted","Data":"fe952605a0f3609f31b1a97c376863e0365a0dcd0dc6f1f8802dc2b089ffec3b"} Nov 02 22:02:19.784009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.782736 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" podStartSLOduration=1.782722117 podStartE2EDuration="1.782722117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:19.780162744 +0000 UTC m=+2969.335791303" watchObservedRunningTime="2026-11-02 22:02:19.782722117 +0000 UTC m=+2969.338350482" Nov 02 22:02:19.857376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.857353 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:19.891310 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.891253 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" podStartSLOduration=1.891235129 podStartE2EDuration="1.891235129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:19.886747459 +0000 UTC m=+2969.442375813" watchObservedRunningTime="2026-11-02 22:02:19.891235129 +0000 UTC m=+2969.446863470" Nov 02 22:02:19.954471 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.954402 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-xks-ca\") pod \"47d15082-8d63-409b-bc2c-fc2159d7cffa\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " Nov 02 22:02:19.954471 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.954441 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-openshift-ca\") pod \"47d15082-8d63-409b-bc2c-fc2159d7cffa\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " Nov 02 22:02:19.954471 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.954468 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d15082-8d63-409b-bc2c-fc2159d7cffa-kube-api-access-5p2lr\" (UniqueName: \"kubernetes.io/projected/47d15082-8d63-409b-bc2c-fc2159d7cffa-kube-api-access-5p2lr\") pod \"47d15082-8d63-409b-bc2c-fc2159d7cffa\" (UID: \"47d15082-8d63-409b-bc2c-fc2159d7cffa\") " Nov 02 22:02:19.954869 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.954805 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-openshift-ca" pod "47d15082-8d63-409b-bc2c-fc2159d7cffa" (UID: "47d15082-8d63-409b-bc2c-fc2159d7cffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:19.954911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.954879 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:19.956504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.956475 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-xks-ca" pod "47d15082-8d63-409b-bc2c-fc2159d7cffa" (UID: "47d15082-8d63-409b-bc2c-fc2159d7cffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:19.956630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:19.956597 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d15082-8d63-409b-bc2c-fc2159d7cffa-kube-api-access-5p2lr" pod "47d15082-8d63-409b-bc2c-fc2159d7cffa" (UID: "47d15082-8d63-409b-bc2c-fc2159d7cffa"). InnerVolumeSpecName "kube-api-access-5p2lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:20.055709 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.055675 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d15082-8d63-409b-bc2c-fc2159d7cffa-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:20.055709 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.055704 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2lr\" (UniqueName: \"kubernetes.io/projected/47d15082-8d63-409b-bc2c-fc2159d7cffa-kube-api-access-5p2lr\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:20.300034 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.299932 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:20.683661 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.683632 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" Nov 02 22:02:20.683824 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.683672 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5" event={"ID":"47d15082-8d63-409b-bc2c-fc2159d7cffa","Type":"ContainerDied","Data":"a926376805939d05ee5addc425036cb38a0375c6f923f792ae05387beecf5c4c"} Nov 02 22:02:20.683824 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.683726 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a926376805939d05ee5addc425036cb38a0375c6f923f792ae05387beecf5c4c" Nov 02 22:02:20.685797 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.685770 4441 generic.go:356] "Generic (PLEG): container finished" podID="e5f37c3c-f91e-43c8-a8f2-33335d721702" containerID="7e1cad03fe0e7449226abb1a499a3cad43da78ebb8d5a7099f59c46c3a2ef9d1" exitCode=0 Nov 02 22:02:20.685939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.685876 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" event={"ID":"e5f37c3c-f91e-43c8-a8f2-33335d721702","Type":"ContainerDied","Data":"7e1cad03fe0e7449226abb1a499a3cad43da78ebb8d5a7099f59c46c3a2ef9d1"} Nov 02 22:02:20.687817 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.687795 4441 generic.go:356] "Generic (PLEG): container finished" podID="a976d014-017a-4cf3-bbf0-eec2fce0da38" containerID="7759f2e24ba9d2a33404ddf1f094157f77b89ce62279a4ed1fa6f350f4f25719" exitCode=0 Nov 02 22:02:20.687923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:20.687903 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" event={"ID":"a976d014-017a-4cf3-bbf0-eec2fce0da38","Type":"ContainerDied","Data":"7759f2e24ba9d2a33404ddf1f094157f77b89ce62279a4ed1fa6f350f4f25719"} Nov 02 22:02:21.867948 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.867687 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:21.872110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.872090 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:21.977130 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.977054 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-openshift-ca\") pod \"a976d014-017a-4cf3-bbf0-eec2fce0da38\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " Nov 02 22:02:21.977597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.977135 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a976d014-017a-4cf3-bbf0-eec2fce0da38-kube-api-access-tzl8m\" (UniqueName: \"kubernetes.io/projected/a976d014-017a-4cf3-bbf0-eec2fce0da38-kube-api-access-tzl8m\") pod \"a976d014-017a-4cf3-bbf0-eec2fce0da38\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " Nov 02 22:02:21.977597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.977226 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-xks-ca\") pod \"a976d014-017a-4cf3-bbf0-eec2fce0da38\" (UID: \"a976d014-017a-4cf3-bbf0-eec2fce0da38\") " Nov 02 22:02:21.977597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.977272 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-xks-ca\") pod \"e5f37c3c-f91e-43c8-a8f2-33335d721702\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " Nov 02 22:02:21.977597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.977375 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f37c3c-f91e-43c8-a8f2-33335d721702-kube-api-access-wbj9z\" (UniqueName: \"kubernetes.io/projected/e5f37c3c-f91e-43c8-a8f2-33335d721702-kube-api-access-wbj9z\") pod \"e5f37c3c-f91e-43c8-a8f2-33335d721702\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " Nov 02 22:02:21.977597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.977417 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-openshift-ca\") pod \"e5f37c3c-f91e-43c8-a8f2-33335d721702\" (UID: \"e5f37c3c-f91e-43c8-a8f2-33335d721702\") " Nov 02 22:02:21.978546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.978384 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-openshift-ca" pod "e5f37c3c-f91e-43c8-a8f2-33335d721702" (UID: "e5f37c3c-f91e-43c8-a8f2-33335d721702"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:21.978782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.978686 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-openshift-ca" pod "a976d014-017a-4cf3-bbf0-eec2fce0da38" (UID: "a976d014-017a-4cf3-bbf0-eec2fce0da38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:21.983884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.983862 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a976d014-017a-4cf3-bbf0-eec2fce0da38-kube-api-access-tzl8m" pod "a976d014-017a-4cf3-bbf0-eec2fce0da38" (UID: "a976d014-017a-4cf3-bbf0-eec2fce0da38"). InnerVolumeSpecName "kube-api-access-tzl8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:21.987353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.987316 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-xks-ca" pod "e5f37c3c-f91e-43c8-a8f2-33335d721702" (UID: "e5f37c3c-f91e-43c8-a8f2-33335d721702"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:21.987715 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.987687 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f37c3c-f91e-43c8-a8f2-33335d721702-kube-api-access-wbj9z" pod "e5f37c3c-f91e-43c8-a8f2-33335d721702" (UID: "e5f37c3c-f91e-43c8-a8f2-33335d721702"). InnerVolumeSpecName "kube-api-access-wbj9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:21.995093 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:21.995058 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-xks-ca" pod "a976d014-017a-4cf3-bbf0-eec2fce0da38" (UID: "a976d014-017a-4cf3-bbf0-eec2fce0da38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:22.078527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.078494 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.078527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.078526 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzl8m\" (UniqueName: \"kubernetes.io/projected/a976d014-017a-4cf3-bbf0-eec2fce0da38-kube-api-access-tzl8m\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.078727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.078541 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a976d014-017a-4cf3-bbf0-eec2fce0da38-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.078727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.078557 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.078727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.078571 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbj9z\" (UniqueName: \"kubernetes.io/projected/e5f37c3c-f91e-43c8-a8f2-33335d721702-kube-api-access-wbj9z\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.078727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.078585 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f37c3c-f91e-43c8-a8f2-33335d721702-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.706392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.706347 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" event={"ID":"e5f37c3c-f91e-43c8-a8f2-33335d721702","Type":"ContainerDied","Data":"ad063b9c08ee440f59ebfff946f65109d6e4d097a589351b06f4a373be10d7c8"} Nov 02 22:02:22.706574 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.706419 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad063b9c08ee440f59ebfff946f65109d6e4d097a589351b06f4a373be10d7c8" Nov 02 22:02:22.706574 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.706370 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m" Nov 02 22:02:22.708311 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.708037 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" Nov 02 22:02:22.708311 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.708109 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4" event={"ID":"a976d014-017a-4cf3-bbf0-eec2fce0da38","Type":"ContainerDied","Data":"fe952605a0f3609f31b1a97c376863e0365a0dcd0dc6f1f8802dc2b089ffec3b"} Nov 02 22:02:22.708311 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:22.708181 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe952605a0f3609f31b1a97c376863e0365a0dcd0dc6f1f8802dc2b089ffec3b" Nov 02 22:02:23.758729 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:23.758695 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6"] Nov 02 22:02:23.860430 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:23.860400 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn"] Nov 02 22:02:24.100123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:24.100089 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:24.222758 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:24.222726 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:25.105561 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:25.105522 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:27.692313 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:27.692269 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:28.759029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:28.758986 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" podUID="fbffaf2e-9895-426c-85db-11c650e7306d" containerName="payload-processing" containerID="cri-o://d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79" gracePeriod=25 Nov 02 22:02:28.861297 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:28.861252 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" podUID="dcb6c070-74c2-4dd1-9173-28e6020c3142" containerName="payload-pre-processing" containerID="cri-o://b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9" gracePeriod=25 Nov 02 22:02:29.062854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.062824 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 22:02:29.169078 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.169049 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbffaf2e-9895-426c-85db-11c650e7306d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbffaf2e-9895-426c-85db-11c650e7306d-plugins-config-volume\") pod \"fbffaf2e-9895-426c-85db-11c650e7306d\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " Nov 02 22:02:29.169217 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.169205 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbffaf2e-9895-426c-85db-11c650e7306d-kube-api-access-hctjl\" (UniqueName: \"kubernetes.io/projected/fbffaf2e-9895-426c-85db-11c650e7306d-kube-api-access-hctjl\") pod \"fbffaf2e-9895-426c-85db-11c650e7306d\" (UID: \"fbffaf2e-9895-426c-85db-11c650e7306d\") " Nov 02 22:02:29.169376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.169354 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbffaf2e-9895-426c-85db-11c650e7306d-plugins-config-volume" pod "fbffaf2e-9895-426c-85db-11c650e7306d" (UID: "fbffaf2e-9895-426c-85db-11c650e7306d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:29.169548 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.169535 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbffaf2e-9895-426c-85db-11c650e7306d-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:29.171452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.171430 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 22:02:29.172337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.171571 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbffaf2e-9895-426c-85db-11c650e7306d-kube-api-access-hctjl" pod "fbffaf2e-9895-426c-85db-11c650e7306d" (UID: "fbffaf2e-9895-426c-85db-11c650e7306d"). InnerVolumeSpecName "kube-api-access-hctjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:29.269933 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.269900 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb6c070-74c2-4dd1-9173-28e6020c3142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb6c070-74c2-4dd1-9173-28e6020c3142-plugins-config-volume\") pod \"dcb6c070-74c2-4dd1-9173-28e6020c3142\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " Nov 02 22:02:29.270101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.269949 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb6c070-74c2-4dd1-9173-28e6020c3142-kube-api-access-td7fc\" (UniqueName: \"kubernetes.io/projected/dcb6c070-74c2-4dd1-9173-28e6020c3142-kube-api-access-td7fc\") pod \"dcb6c070-74c2-4dd1-9173-28e6020c3142\" (UID: \"dcb6c070-74c2-4dd1-9173-28e6020c3142\") " Nov 02 22:02:29.270273 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.270257 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hctjl\" (UniqueName: \"kubernetes.io/projected/fbffaf2e-9895-426c-85db-11c650e7306d-kube-api-access-hctjl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:29.270367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.270344 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb6c070-74c2-4dd1-9173-28e6020c3142-plugins-config-volume" pod "dcb6c070-74c2-4dd1-9173-28e6020c3142" (UID: "dcb6c070-74c2-4dd1-9173-28e6020c3142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:29.272166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.272141 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb6c070-74c2-4dd1-9173-28e6020c3142-kube-api-access-td7fc" pod "dcb6c070-74c2-4dd1-9173-28e6020c3142" (UID: "dcb6c070-74c2-4dd1-9173-28e6020c3142"). InnerVolumeSpecName "kube-api-access-td7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:29.371265 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.371228 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb6c070-74c2-4dd1-9173-28e6020c3142-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:29.371265 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.371258 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td7fc\" (UniqueName: \"kubernetes.io/projected/dcb6c070-74c2-4dd1-9173-28e6020c3142-kube-api-access-td7fc\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:29.473161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.473108 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:29.621943 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.621850 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:29.772299 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.772264 4441 generic.go:356] "Generic (PLEG): container finished" podID="fbffaf2e-9895-426c-85db-11c650e7306d" containerID="d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79" exitCode=0 Nov 02 22:02:29.772772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.772322 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" Nov 02 22:02:29.772772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.772347 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" event={"ID":"fbffaf2e-9895-426c-85db-11c650e7306d","Type":"ContainerDied","Data":"d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79"} Nov 02 22:02:29.772772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.772399 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6" event={"ID":"fbffaf2e-9895-426c-85db-11c650e7306d","Type":"ContainerDied","Data":"1303971963b0ca0dda452322c83dda3bbbde6d2a6122aa6d0160be024c249235"} Nov 02 22:02:29.772772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.772421 4441 scope.go:122] "RemoveContainer" containerID="d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79" Nov 02 22:02:29.773997 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.773961 4441 generic.go:356] "Generic (PLEG): container finished" podID="dcb6c070-74c2-4dd1-9173-28e6020c3142" containerID="b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9" exitCode=0 Nov 02 22:02:29.774115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.774055 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" Nov 02 22:02:29.774204 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.774182 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" event={"ID":"dcb6c070-74c2-4dd1-9173-28e6020c3142","Type":"ContainerDied","Data":"b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9"} Nov 02 22:02:29.774281 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.774233 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn" event={"ID":"dcb6c070-74c2-4dd1-9173-28e6020c3142","Type":"ContainerDied","Data":"33908d073f6891413d37d39beeb55e1101c44edf9f5f58dad596c5416a683311"} Nov 02 22:02:29.786247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.786229 4441 scope.go:122] "RemoveContainer" containerID="d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79" Nov 02 22:02:29.786511 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:29.786490 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79\": container with ID starting with d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79 not found: ID does not exist" containerID="d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79" Nov 02 22:02:29.786586 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.786515 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79"} err="failed to get container status \"d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79\": rpc error: code = NotFound desc = could not find container \"d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79\": container with ID starting with d0d5747655235018ec7bb65f780b35a545d2a588e6d2c0da0432f4a5cf474b79 not found: ID does not exist" Nov 02 22:02:29.786586 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.786531 4441 scope.go:122] "RemoveContainer" containerID="b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9" Nov 02 22:02:29.799895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.799099 4441 scope.go:122] "RemoveContainer" containerID="b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9" Nov 02 22:02:29.799992 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:29.799956 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9\": container with ID starting with b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9 not found: ID does not exist" containerID="b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9" Nov 02 22:02:29.800056 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.799999 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9"} err="failed to get container status \"b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9\": rpc error: code = NotFound desc = could not find container \"b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9\": container with ID starting with b772922ddce36ddeb3f456a2391922f4b8db68cfce9db4f67f8e7cdffa52cef9 not found: ID does not exist" Nov 02 22:02:29.930641 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.930571 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6"] Nov 02 22:02:29.951485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.951426 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:29.977306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:29.977279 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f4a242-66587fdf75-qghj6"] Nov 02 22:02:30.082599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.082565 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:30.082849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.082828 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn"] Nov 02 22:02:30.087122 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.087099 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f4a242-6cd88bd755-lkvsn"] Nov 02 22:02:30.220919 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.220848 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:30.584808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.584775 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb6c070-74c2-4dd1-9173-28e6020c3142" path="/var/lib/kubelet/pods/dcb6c070-74c2-4dd1-9173-28e6020c3142/volumes" Nov 02 22:02:30.585186 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.585171 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbffaf2e-9895-426c-85db-11c650e7306d" path="/var/lib/kubelet/pods/fbffaf2e-9895-426c-85db-11c650e7306d/volumes" Nov 02 22:02:30.893749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.893678 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv"] Nov 02 22:02:30.894338 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894320 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a976d014-017a-4cf3-bbf0-eec2fce0da38" containerName="revoke-keys" Nov 02 22:02:30.894393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894340 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a976d014-017a-4cf3-bbf0-eec2fce0da38" containerName="revoke-keys" Nov 02 22:02:30.894393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894351 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e5f37c3c-f91e-43c8-a8f2-33335d721702" containerName="revoke-keys" Nov 02 22:02:30.894393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894356 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f37c3c-f91e-43c8-a8f2-33335d721702" containerName="revoke-keys" Nov 02 22:02:30.894393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894375 4441 cpu_manager.go:414] "containerMap: removing container" podUID="47d15082-8d63-409b-bc2c-fc2159d7cffa" containerName="revoke-keys" Nov 02 22:02:30.894393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894380 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d15082-8d63-409b-bc2c-fc2159d7cffa" containerName="revoke-keys" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894414 4441 cpu_manager.go:414] "containerMap: removing container" podUID="fbffaf2e-9895-426c-85db-11c650e7306d" containerName="payload-processing" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894419 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbffaf2e-9895-426c-85db-11c650e7306d" containerName="payload-processing" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894429 4441 cpu_manager.go:414] "containerMap: removing container" podUID="dcb6c070-74c2-4dd1-9173-28e6020c3142" containerName="payload-pre-processing" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894434 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb6c070-74c2-4dd1-9173-28e6020c3142" containerName="payload-pre-processing" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894526 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a976d014-017a-4cf3-bbf0-eec2fce0da38" containerName="revoke-keys" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894535 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f37c3c-f91e-43c8-a8f2-33335d721702" containerName="revoke-keys" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894544 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d15082-8d63-409b-bc2c-fc2159d7cffa" containerName="revoke-keys" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894554 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbffaf2e-9895-426c-85db-11c650e7306d" containerName="payload-processing" Nov 02 22:02:30.894557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.894560 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb6c070-74c2-4dd1-9173-28e6020c3142" containerName="payload-pre-processing" Nov 02 22:02:30.899778 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.899759 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.978880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.978848 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv"] Nov 02 22:02:30.987470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987440 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtsc\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-kube-api-access-gdtsc\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.987633 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987482 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-credential-socket\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.987633 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987508 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-envoy\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.987633 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987575 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b2255-648f-4abe-91f6-d29770490815-istiod-ca-cert\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.987835 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987640 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-certs\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.987901 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987839 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-data\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.987960 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987894 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b2255-648f-4abe-91f6-d29770490815-istio-podinfo\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.987960 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987927 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-socket\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:30.988103 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:30.987990 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-istio-token\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.002445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.002422 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-49c06d-openshift-default-dockercfg-nx846\"" Nov 02 22:02:31.084400 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.084362 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:31.089714 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089688 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-data\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.089852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089729 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b2255-648f-4abe-91f6-d29770490815-istio-podinfo\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.089852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089751 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-socket\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.089852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089776 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-istio-token\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.089852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089810 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtsc\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-kube-api-access-gdtsc\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.089852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089834 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-credential-socket\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.089852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089849 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-envoy\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.090168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089872 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b2255-648f-4abe-91f6-d29770490815-istiod-ca-cert\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.090168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.089936 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-certs\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.090278 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.090175 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-data\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.090278 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.090203 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-credential-socket\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.090411 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.090377 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-certs\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.090490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.090468 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-socket\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.090778 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.090761 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b2255-648f-4abe-91f6-d29770490815-istiod-ca-cert\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.092242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.092216 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-envoy\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.092622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.092604 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b2255-648f-4abe-91f6-d29770490815-istio-podinfo\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.143867 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.143794 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-istio-token\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.144059 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.143939 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtsc\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-kube-api-access-gdtsc\") pod \"e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.216511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.216478 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:31.217239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.217191 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:31.479600 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.479510 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:31.634830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.634798 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv"] Nov 02 22:02:31.635509 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:31.635481 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801b2255_648f_4abe_91f6_d29770490815.slice/crio-19aff9c0912db8b95a2c7192b72b3229d0468f47d4825f2bbbbcb54dc646e839 WatchSource:0}: Error finding container 19aff9c0912db8b95a2c7192b72b3229d0468f47d4825f2bbbbcb54dc646e839: Status 404 returned error can't find the container with id 19aff9c0912db8b95a2c7192b72b3229d0468f47d4825f2bbbbcb54dc646e839 Nov 02 22:02:31.638437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.638409 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:02:31.638545 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.638503 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:02:31.638617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.638550 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:02:31.794528 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.794314 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" event={"ID":"801b2255-648f-4abe-91f6-d29770490815","Type":"ContainerStarted","Data":"4c00219cd4fa9c1647445ceb190b25981255c0b09d2ba0a319eb7cb6df54677f"} Nov 02 22:02:31.794528 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.794506 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" event={"ID":"801b2255-648f-4abe-91f6-d29770490815","Type":"ContainerStarted","Data":"19aff9c0912db8b95a2c7192b72b3229d0468f47d4825f2bbbbcb54dc646e839"} Nov 02 22:02:31.907007 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:31.906945 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" podStartSLOduration=1.906930661 podStartE2EDuration="1.906930661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:31.903687124 +0000 UTC m=+2981.459315502" watchObservedRunningTime="2026-11-02 22:02:31.906930661 +0000 UTC m=+2981.462559043" Nov 02 22:02:32.216882 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:32.216845 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:32.223181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:32.223149 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:32.803997 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:32.803939 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:32.805041 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:32.805022 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:02:34.390796 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:34.390760 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:34.649192 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:34.649099 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:34.773385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:34.773339 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:34.884668 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:34.884631 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4"] Nov 02 22:02:34.929125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:34.929052 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11537e-xk6m4"] Nov 02 22:02:35.237301 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:35.237217 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:35.387162 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:35.387127 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:35.522690 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:35.522609 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:35.614957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:35.614925 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh"] Nov 02 22:02:35.650626 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:35.650594 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:36.116342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.116306 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w"] Nov 02 22:02:36.120498 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.120476 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.173442 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.173418 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w"] Nov 02 22:02:36.215659 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.215627 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:02:36.215901 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.215627 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:02:36.215901 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.215641 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:02:36.215901 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.215875 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:02:36.252256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.252224 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.252444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.252419 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.252527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.252463 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/ba4c7c37-3ea7-4393-b6c0-79243a316c64-kube-api-access-9hp7t\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.354024 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.353959 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.354185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.354035 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/ba4c7c37-3ea7-4393-b6c0-79243a316c64-kube-api-access-9hp7t\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.354185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.354095 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.354759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.354724 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.356269 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.356241 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.411831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.411759 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/ba4c7c37-3ea7-4393-b6c0-79243a316c64-kube-api-access-9hp7t\") pod \"maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.434351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.434325 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:36.449806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.449780 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:36.591510 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.591478 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a976d014-017a-4cf3-bbf0-eec2fce0da38" path="/var/lib/kubelet/pods/a976d014-017a-4cf3-bbf0-eec2fce0da38/volumes" Nov 02 22:02:36.804082 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:36.804035 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4c7c37_3ea7_4393_b6c0_79243a316c64.slice/crio-be1cce5beac969237250cb48f30308de5b827f1de5a7744e7f737ee34cde5e16 WatchSource:0}: Error finding container be1cce5beac969237250cb48f30308de5b827f1de5a7744e7f737ee34cde5e16: Status 404 returned error can't find the container with id be1cce5beac969237250cb48f30308de5b827f1de5a7744e7f737ee34cde5e16 Nov 02 22:02:36.804233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.804128 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w"] Nov 02 22:02:36.850077 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:36.850050 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" event={"ID":"ba4c7c37-3ea7-4393-b6c0-79243a316c64","Type":"ContainerStarted","Data":"be1cce5beac969237250cb48f30308de5b827f1de5a7744e7f737ee34cde5e16"} Nov 02 22:02:37.858395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:37.858356 4441 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7c37-3ea7-4393-b6c0-79243a316c64" containerID="e48a4310e1fc1b7253c4efcc5ca2da2ffc865b78a45faf3120558e84502eed7e" exitCode=0 Nov 02 22:02:37.858835 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:37.858445 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" event={"ID":"ba4c7c37-3ea7-4393-b6c0-79243a316c64","Type":"ContainerDied","Data":"e48a4310e1fc1b7253c4efcc5ca2da2ffc865b78a45faf3120558e84502eed7e"} Nov 02 22:02:39.022149 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.022121 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:39.085538 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.085503 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-openshift-ca\") pod \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " Nov 02 22:02:39.085685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.085636 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4c7c37-3ea7-4393-b6c0-79243a316c64-kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/ba4c7c37-3ea7-4393-b6c0-79243a316c64-kube-api-access-9hp7t\") pod \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " Nov 02 22:02:39.085804 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.085786 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-xks-ca\") pod \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\" (UID: \"ba4c7c37-3ea7-4393-b6c0-79243a316c64\") " Nov 02 22:02:39.085898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.085877 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-openshift-ca" pod "ba4c7c37-3ea7-4393-b6c0-79243a316c64" (UID: "ba4c7c37-3ea7-4393-b6c0-79243a316c64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:39.086243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.086221 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:39.087787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.087751 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4c7c37-3ea7-4393-b6c0-79243a316c64-kube-api-access-9hp7t" pod "ba4c7c37-3ea7-4393-b6c0-79243a316c64" (UID: "ba4c7c37-3ea7-4393-b6c0-79243a316c64"). InnerVolumeSpecName "kube-api-access-9hp7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:39.087921 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.087817 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-xks-ca" pod "ba4c7c37-3ea7-4393-b6c0-79243a316c64" (UID: "ba4c7c37-3ea7-4393-b6c0-79243a316c64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:39.187665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.187575 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4c7c37-3ea7-4393-b6c0-79243a316c64-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:39.187665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.187614 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/ba4c7c37-3ea7-4393-b6c0-79243a316c64-kube-api-access-9hp7t\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:39.877429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.877399 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" Nov 02 22:02:39.877638 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.877394 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w" event={"ID":"ba4c7c37-3ea7-4393-b6c0-79243a316c64","Type":"ContainerDied","Data":"be1cce5beac969237250cb48f30308de5b827f1de5a7744e7f737ee34cde5e16"} Nov 02 22:02:39.877638 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:39.877524 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be1cce5beac969237250cb48f30308de5b827f1de5a7744e7f737ee34cde5e16" Nov 02 22:02:40.617132 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:40.617093 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:02:40.926809 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:40.926716 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:41.098436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.098401 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:41.343242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.343208 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf"] Nov 02 22:02:41.343776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.343760 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ba4c7c37-3ea7-4393-b6c0-79243a316c64" containerName="revoke-keys" Nov 02 22:02:41.343841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.343777 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4c7c37-3ea7-4393-b6c0-79243a316c64" containerName="revoke-keys" Nov 02 22:02:41.343900 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.343890 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4c7c37-3ea7-4393-b6c0-79243a316c64" containerName="revoke-keys" Nov 02 22:02:41.347425 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.347398 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.353278 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.353242 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:41.400857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.400822 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf"] Nov 02 22:02:41.442641 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.442504 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-49c06d-dockercfg-nr44h\"" Nov 02 22:02:41.442641 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.442611 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-49c06d\"" Nov 02 22:02:41.447740 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.447711 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150790a2-98b0-44ba-ab92-06ab0a0311dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.447935 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.447899 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8h4\" (UniqueName: \"kubernetes.io/projected/150790a2-98b0-44ba-ab92-06ab0a0311dd-kube-api-access-zz8h4\") pod \"payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.510016 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.509981 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v"] Nov 02 22:02:41.514072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.514049 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.549474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.549447 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8h4\" (UniqueName: \"kubernetes.io/projected/150790a2-98b0-44ba-ab92-06ab0a0311dd-kube-api-access-zz8h4\") pod \"payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.549608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.549563 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150790a2-98b0-44ba-ab92-06ab0a0311dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.550256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.550233 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150790a2-98b0-44ba-ab92-06ab0a0311dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.553471 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.553451 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:41.567163 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.567141 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v"] Nov 02 22:02:41.604481 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.604429 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8h4\" (UniqueName: \"kubernetes.io/projected/150790a2-98b0-44ba-ab92-06ab0a0311dd-kube-api-access-zz8h4\") pod \"payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.651087 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.651057 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxf6\" (UniqueName: \"kubernetes.io/projected/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-kube-api-access-sdxf6\") pod \"payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.651458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.651159 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.667904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.667875 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:41.690856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.690832 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:41.752404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.752375 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxf6\" (UniqueName: \"kubernetes.io/projected/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-kube-api-access-sdxf6\") pod \"payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.752528 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.752439 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.753201 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.753180 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.806282 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.806250 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxf6\" (UniqueName: \"kubernetes.io/projected/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-kube-api-access-sdxf6\") pod \"payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.827924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.827724 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:41.858541 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.858466 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw"] Nov 02 22:02:41.858697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.858674 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" containerID="cri-o://3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7" gracePeriod=30 Nov 02 22:02:41.882314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:41.882285 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:42.041053 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.041018 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf"] Nov 02 22:02:42.041345 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:42.041322 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150790a2_98b0_44ba_ab92_06ab0a0311dd.slice/crio-a622365c7ddc4077de389619a5bfa0815c487d5bb438fc4f82364828e8105176 WatchSource:0}: Error finding container a622365c7ddc4077de389619a5bfa0815c487d5bb438fc4f82364828e8105176: Status 404 returned error can't find the container with id a622365c7ddc4077de389619a5bfa0815c487d5bb438fc4f82364828e8105176 Nov 02 22:02:42.211964 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.211929 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v"] Nov 02 22:02:42.213574 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:42.213546 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc03a9eb_98bb_48dd_b05b_21d3f8d43ccd.slice/crio-29181a07394ca1856404329c68f76185be2c18d4048d13608923bcc58af5ecdc WatchSource:0}: Error finding container 29181a07394ca1856404329c68f76185be2c18d4048d13608923bcc58af5ecdc: Status 404 returned error can't find the container with id 29181a07394ca1856404329c68f76185be2c18d4048d13608923bcc58af5ecdc Nov 02 22:02:42.314112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.314075 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:42.785830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.785790 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr"] Nov 02 22:02:42.790049 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.790029 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.839831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.839800 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:02:42.840022 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.839900 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:02:42.840022 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.839930 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:02:42.840152 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.840123 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:02:42.842608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.842585 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr"] Nov 02 22:02:42.871384 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.871356 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.871501 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.871415 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.871581 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.871556 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdhq\" (UniqueName: \"kubernetes.io/projected/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-kube-api-access-lzdhq\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.910931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.910897 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" event={"ID":"150790a2-98b0-44ba-ab92-06ab0a0311dd","Type":"ContainerStarted","Data":"74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4"} Nov 02 22:02:42.911099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.910953 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" event={"ID":"150790a2-98b0-44ba-ab92-06ab0a0311dd","Type":"ContainerStarted","Data":"a622365c7ddc4077de389619a5bfa0815c487d5bb438fc4f82364828e8105176"} Nov 02 22:02:42.911179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.911118 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:42.912751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.912725 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" event={"ID":"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd","Type":"ContainerStarted","Data":"a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad"} Nov 02 22:02:42.912877 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.912773 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" event={"ID":"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd","Type":"ContainerStarted","Data":"29181a07394ca1856404329c68f76185be2c18d4048d13608923bcc58af5ecdc"} Nov 02 22:02:42.912877 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.912832 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:42.972740 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.972719 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdhq\" (UniqueName: \"kubernetes.io/projected/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-kube-api-access-lzdhq\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.973045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.973025 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.973137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.973064 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.973721 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.973700 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:42.975252 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:42.975232 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:43.019739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:43.019698 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" podStartSLOduration=2.01968757 podStartE2EDuration="2.01968757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:43.017425249 +0000 UTC m=+2992.573053597" watchObservedRunningTime="2026-11-02 22:02:43.01968757 +0000 UTC m=+2992.575315906" Nov 02 22:02:43.028460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:43.028432 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdhq\" (UniqueName: \"kubernetes.io/projected/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-kube-api-access-lzdhq\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:43.104917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:43.104891 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:43.129219 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:43.129149 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" podStartSLOduration=2.129131663 podStartE2EDuration="2.129131663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:43.126554646 +0000 UTC m=+2992.682182984" watchObservedRunningTime="2026-11-02 22:02:43.129131663 +0000 UTC m=+2992.684760001" Nov 02 22:02:43.479850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:43.479819 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr"] Nov 02 22:02:43.480016 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:43.479990 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b302d04_e3f2_4c17_8f08_e1939d74f8e0.slice/crio-16fd7c49656ed1bc56c079966d859dffb7c27802ee430a0acec3ca371b4fef67 WatchSource:0}: Error finding container 16fd7c49656ed1bc56c079966d859dffb7c27802ee430a0acec3ca371b4fef67: Status 404 returned error can't find the container with id 16fd7c49656ed1bc56c079966d859dffb7c27802ee430a0acec3ca371b4fef67 Nov 02 22:02:43.921774 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:43.921740 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" event={"ID":"3b302d04-e3f2-4c17-8f08-e1939d74f8e0","Type":"ContainerStarted","Data":"41b169cf645b2ea39428a4a7ad9e2cef8db6eae0b01300c0ceb6df49fd8b97c7"} Nov 02 22:02:43.922245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:43.921796 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" event={"ID":"3b302d04-e3f2-4c17-8f08-e1939d74f8e0","Type":"ContainerStarted","Data":"16fd7c49656ed1bc56c079966d859dffb7c27802ee430a0acec3ca371b4fef67"} Nov 02 22:02:44.028552 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.028495 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" podStartSLOduration=2.028476026 podStartE2EDuration="2.028476026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:44.027908983 +0000 UTC m=+2993.583537320" watchObservedRunningTime="2026-11-02 22:02:44.028476026 +0000 UTC m=+2993.584104412" Nov 02 22:02:44.349635 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.349605 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:44.803059 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.802960 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x"] Nov 02 22:02:44.807084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.807065 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:44.860145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.860120 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x"] Nov 02 22:02:44.897000 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.896948 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42bg\" (UniqueName: \"kubernetes.io/projected/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-kube-api-access-b42bg\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:44.897142 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.897063 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:44.897270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.897253 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:44.930149 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.930123 4441 generic.go:356] "Generic (PLEG): container finished" podID="3b302d04-e3f2-4c17-8f08-e1939d74f8e0" containerID="41b169cf645b2ea39428a4a7ad9e2cef8db6eae0b01300c0ceb6df49fd8b97c7" exitCode=0 Nov 02 22:02:44.930478 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.930209 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" event={"ID":"3b302d04-e3f2-4c17-8f08-e1939d74f8e0","Type":"ContainerDied","Data":"41b169cf645b2ea39428a4a7ad9e2cef8db6eae0b01300c0ceb6df49fd8b97c7"} Nov 02 22:02:44.997830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.997804 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:44.997947 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.997845 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42bg\" (UniqueName: \"kubernetes.io/projected/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-kube-api-access-b42bg\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:44.997947 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.997879 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:44.998523 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:44.998501 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:45.000186 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:45.000164 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:45.054395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:45.054317 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42bg\" (UniqueName: \"kubernetes.io/projected/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-kube-api-access-b42bg\") pod \"maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:45.120958 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:45.120927 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:45.492781 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:02:45.492744 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf084e9bd_2791_43cb_b8df_dfd56dc51b9e.slice/crio-63dd497d23c573ae91ea91a7c4e6c233d4d0b82cb16bb32815b40da2fcbb4384 WatchSource:0}: Error finding container 63dd497d23c573ae91ea91a7c4e6c233d4d0b82cb16bb32815b40da2fcbb4384: Status 404 returned error can't find the container with id 63dd497d23c573ae91ea91a7c4e6c233d4d0b82cb16bb32815b40da2fcbb4384 Nov 02 22:02:45.494071 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:45.494040 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x"] Nov 02 22:02:45.939144 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:45.939101 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" event={"ID":"f084e9bd-2791-43cb-b8df-dfd56dc51b9e","Type":"ContainerStarted","Data":"4c630bdf0e83e29976a494c9eb6aacd9a7d87931eba65e3c1ca6b14b8f6b30cb"} Nov 02 22:02:45.939618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:45.939157 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" event={"ID":"f084e9bd-2791-43cb-b8df-dfd56dc51b9e","Type":"ContainerStarted","Data":"63dd497d23c573ae91ea91a7c4e6c233d4d0b82cb16bb32815b40da2fcbb4384"} Nov 02 22:02:46.050782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.050682 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" podStartSLOduration=2.050668583 podStartE2EDuration="2.050668583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:46.047302992 +0000 UTC m=+2995.602931330" watchObservedRunningTime="2026-11-02 22:02:46.050668583 +0000 UTC m=+2995.606296980" Nov 02 22:02:46.085724 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.085700 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:46.112189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.112164 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-kube-api-access-lzdhq\" (UniqueName: \"kubernetes.io/projected/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-kube-api-access-lzdhq\") pod \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " Nov 02 22:02:46.112343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.112330 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-openshift-ca\") pod \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " Nov 02 22:02:46.112454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.112442 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-xks-ca\") pod \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\" (UID: \"3b302d04-e3f2-4c17-8f08-e1939d74f8e0\") " Nov 02 22:02:46.112666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.112643 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-openshift-ca" pod "3b302d04-e3f2-4c17-8f08-e1939d74f8e0" (UID: "3b302d04-e3f2-4c17-8f08-e1939d74f8e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:46.114111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.114092 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-kube-api-access-lzdhq" pod "3b302d04-e3f2-4c17-8f08-e1939d74f8e0" (UID: "3b302d04-e3f2-4c17-8f08-e1939d74f8e0"). InnerVolumeSpecName "kube-api-access-lzdhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:46.114346 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.114323 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-xks-ca" pod "3b302d04-e3f2-4c17-8f08-e1939d74f8e0" (UID: "3b302d04-e3f2-4c17-8f08-e1939d74f8e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:46.213495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.213464 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:46.213495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.213492 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:46.213689 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.213504 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzdhq\" (UniqueName: \"kubernetes.io/projected/3b302d04-e3f2-4c17-8f08-e1939d74f8e0-kube-api-access-lzdhq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:46.279097 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.279062 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:46.948032 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.947998 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" event={"ID":"3b302d04-e3f2-4c17-8f08-e1939d74f8e0","Type":"ContainerDied","Data":"16fd7c49656ed1bc56c079966d859dffb7c27802ee430a0acec3ca371b4fef67"} Nov 02 22:02:46.948499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.948059 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16fd7c49656ed1bc56c079966d859dffb7c27802ee430a0acec3ca371b4fef67" Nov 02 22:02:46.948499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.948011 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr" Nov 02 22:02:46.949639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.949610 4441 generic.go:356] "Generic (PLEG): container finished" podID="f084e9bd-2791-43cb-b8df-dfd56dc51b9e" containerID="4c630bdf0e83e29976a494c9eb6aacd9a7d87931eba65e3c1ca6b14b8f6b30cb" exitCode=0 Nov 02 22:02:46.949771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:46.949694 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" event={"ID":"f084e9bd-2791-43cb-b8df-dfd56dc51b9e","Type":"ContainerDied","Data":"4c630bdf0e83e29976a494c9eb6aacd9a7d87931eba65e3c1ca6b14b8f6b30cb"} Nov 02 22:02:47.139991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.139920 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 22:02:47.223948 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.223914 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-podinfo\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224148 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.223996 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-kube-api-access-69kj6\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-kube-api-access-69kj6\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224148 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224093 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-certs\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224148 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224144 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-credential-socket\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224322 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224207 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-socket\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224322 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224247 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e865ee9-3056-48a2-bed0-19b609cb1795-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee9-3056-48a2-bed0-19b609cb1795-istiod-ca-cert\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224322 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224287 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-token\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224324 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-envoy\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224343 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-certs" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:47.224479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224427 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-data\") pod \"3e865ee9-3056-48a2-bed0-19b609cb1795\" (UID: \"3e865ee9-3056-48a2-bed0-19b609cb1795\") " Nov 02 22:02:47.224479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224434 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-credential-socket" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:47.224705 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224505 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-socket" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:47.224705 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224682 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e865ee9-3056-48a2-bed0-19b609cb1795-istiod-ca-cert" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:47.224819 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.224773 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-data" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:47.225314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.225289 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.225435 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.225319 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.225435 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.225335 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.225435 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.225352 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e865ee9-3056-48a2-bed0-19b609cb1795-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.225435 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.225368 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.226481 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.226457 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-podinfo" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:47.226604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.226522 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-envoy" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:47.226669 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.226637 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-kube-api-access-69kj6" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "kube-api-access-69kj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:47.226839 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.226823 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-token" pod "3e865ee9-3056-48a2-bed0-19b609cb1795" (UID: "3e865ee9-3056-48a2-bed0-19b609cb1795"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:47.309500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.309467 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6"] Nov 02 22:02:47.326747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.326718 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.326747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.326743 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kj6\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-kube-api-access-69kj6\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.326747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.326754 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.327045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.326762 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e865ee9-3056-48a2-bed0-19b609cb1795-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:47.401617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.401544 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:47.960369 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.960335 4441 generic.go:356] "Generic (PLEG): container finished" podID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerID="3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7" exitCode=0 Nov 02 22:02:47.960811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.960376 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" event={"ID":"3e865ee9-3056-48a2-bed0-19b609cb1795","Type":"ContainerDied","Data":"3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7"} Nov 02 22:02:47.960811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.960397 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" Nov 02 22:02:47.960811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.960442 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw" event={"ID":"3e865ee9-3056-48a2-bed0-19b609cb1795","Type":"ContainerDied","Data":"00eff63ede474ecf3f59fd1512d7f957f5e926dbd6e5910fe2b0dee275a2a8c0"} Nov 02 22:02:47.960811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.960474 4441 scope.go:122] "RemoveContainer" containerID="3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7" Nov 02 22:02:47.972907 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.972722 4441 scope.go:122] "RemoveContainer" containerID="cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2" Nov 02 22:02:47.985481 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.985455 4441 scope.go:122] "RemoveContainer" containerID="3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7" Nov 02 22:02:47.985769 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:47.985748 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7\": container with ID starting with 3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7 not found: ID does not exist" containerID="3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7" Nov 02 22:02:47.985838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.985777 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7"} err="failed to get container status \"3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7\": rpc error: code = NotFound desc = could not find container \"3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7\": container with ID starting with 3bae2a99a40832e380eca7102a950e5c39047eb206398d419dedf0403c2b6ae7 not found: ID does not exist" Nov 02 22:02:47.985838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.985796 4441 scope.go:122] "RemoveContainer" containerID="cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2" Nov 02 22:02:47.986111 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:47.986091 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2\": container with ID starting with cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2 not found: ID does not exist" containerID="cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2" Nov 02 22:02:47.986182 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:47.986123 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2"} err="failed to get container status \"cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2\": rpc error: code = NotFound desc = could not find container \"cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2\": container with ID starting with cc9c91b49521815c73ae6b7d25ba016a08d1d313e61e51362fffc5cdde1d03c2 not found: ID does not exist" Nov 02 22:02:48.117151 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.117126 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:48.121050 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.121023 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw"] Nov 02 22:02:48.124600 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.124581 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-11537e-openshift-default-7fc77994d5-9hpxw"] Nov 02 22:02:48.235768 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.235683 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-xks-ca\") pod \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " Nov 02 22:02:48.235768 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.235762 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-openshift-ca\") pod \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " Nov 02 22:02:48.236479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.235926 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-kube-api-access-b42bg\" (UniqueName: \"kubernetes.io/projected/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-kube-api-access-b42bg\") pod \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\" (UID: \"f084e9bd-2791-43cb-b8df-dfd56dc51b9e\") " Nov 02 22:02:48.236479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.236142 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-openshift-ca" pod "f084e9bd-2791-43cb-b8df-dfd56dc51b9e" (UID: "f084e9bd-2791-43cb-b8df-dfd56dc51b9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:48.236479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.236418 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:48.238177 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.238139 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-kube-api-access-b42bg" pod "f084e9bd-2791-43cb-b8df-dfd56dc51b9e" (UID: "f084e9bd-2791-43cb-b8df-dfd56dc51b9e"). InnerVolumeSpecName "kube-api-access-b42bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:48.238364 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.238344 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-xks-ca" pod "f084e9bd-2791-43cb-b8df-dfd56dc51b9e" (UID: "f084e9bd-2791-43cb-b8df-dfd56dc51b9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:48.337356 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.337315 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b42bg\" (UniqueName: \"kubernetes.io/projected/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-kube-api-access-b42bg\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:48.337356 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.337351 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f084e9bd-2791-43cb-b8df-dfd56dc51b9e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:48.584634 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.584603 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" path="/var/lib/kubelet/pods/3e865ee9-3056-48a2-bed0-19b609cb1795/volumes" Nov 02 22:02:48.697848 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.697817 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf"] Nov 02 22:02:48.799214 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.799180 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp"] Nov 02 22:02:48.970452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.970371 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" Nov 02 22:02:48.970452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.970413 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x" event={"ID":"f084e9bd-2791-43cb-b8df-dfd56dc51b9e","Type":"ContainerDied","Data":"63dd497d23c573ae91ea91a7c4e6c233d4d0b82cb16bb32815b40da2fcbb4384"} Nov 02 22:02:48.970913 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:48.970459 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63dd497d23c573ae91ea91a7c4e6c233d4d0b82cb16bb32815b40da2fcbb4384" Nov 02 22:02:49.033753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:49.033720 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:49.153731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:49.153691 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:50.635404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.635361 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" podUID="2cdcaf39-4003-4ee2-9886-a1f46cc1857a" containerName="main" containerID="cri-o://0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4" gracePeriod=45 Nov 02 22:02:50.696313 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.696281 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" podUID="2cdcaf39-4003-4ee2-9886-a1f46cc1857a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.243:8000/ready\": dial tcp 10.133.0.243:8000: connect: connection refused" Nov 02 22:02:50.897520 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.897432 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 22:02:50.967221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967194 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tmp-dir\") pod \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " Nov 02 22:02:50.967387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967230 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-kube-api-access-6b842\" (UniqueName: \"kubernetes.io/projected/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-kube-api-access-6b842\") pod \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " Nov 02 22:02:50.967387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967258 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-dshm\") pod \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " Nov 02 22:02:50.967387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967338 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tls-certs\") pod \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " Nov 02 22:02:50.967387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967385 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-home\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-home\") pod \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " Nov 02 22:02:50.967616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967424 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-model-cache\") pod \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\" (UID: \"2cdcaf39-4003-4ee2-9886-a1f46cc1857a\") " Nov 02 22:02:50.967616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967509 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tmp-dir" pod "2cdcaf39-4003-4ee2-9886-a1f46cc1857a" (UID: "2cdcaf39-4003-4ee2-9886-a1f46cc1857a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.967740 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967704 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-home" pod "2cdcaf39-4003-4ee2-9886-a1f46cc1857a" (UID: "2cdcaf39-4003-4ee2-9886-a1f46cc1857a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.967828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967804 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-model-cache" pod "2cdcaf39-4003-4ee2-9886-a1f46cc1857a" (UID: "2cdcaf39-4003-4ee2-9886-a1f46cc1857a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.967999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967953 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.967999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.967989 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.968110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.968004 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.969557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.969537 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-kube-api-access-6b842" pod "2cdcaf39-4003-4ee2-9886-a1f46cc1857a" (UID: "2cdcaf39-4003-4ee2-9886-a1f46cc1857a"). InnerVolumeSpecName "kube-api-access-6b842". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:50.969651 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.969580 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tls-certs" pod "2cdcaf39-4003-4ee2-9886-a1f46cc1857a" (UID: "2cdcaf39-4003-4ee2-9886-a1f46cc1857a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:50.969651 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.969641 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-dshm" pod "2cdcaf39-4003-4ee2-9886-a1f46cc1857a" (UID: "2cdcaf39-4003-4ee2-9886-a1f46cc1857a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.988027 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.988003 4441 generic.go:356] "Generic (PLEG): container finished" podID="2cdcaf39-4003-4ee2-9886-a1f46cc1857a" containerID="0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4" exitCode=0 Nov 02 22:02:50.988106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.988048 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" Nov 02 22:02:50.988106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.988077 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" event={"ID":"2cdcaf39-4003-4ee2-9886-a1f46cc1857a","Type":"ContainerDied","Data":"0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4"} Nov 02 22:02:50.988193 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.988121 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh" event={"ID":"2cdcaf39-4003-4ee2-9886-a1f46cc1857a","Type":"ContainerDied","Data":"5f392495f700fbe86741bc2056a3342df02ce47a3faf180ba8b45c7d0b1e05c1"} Nov 02 22:02:50.988193 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.988139 4441 scope.go:122] "RemoveContainer" containerID="0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4" Nov 02 22:02:50.998643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.998624 4441 scope.go:122] "RemoveContainer" containerID="0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4" Nov 02 22:02:50.998919 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:50.998899 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4\": container with ID starting with 0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4 not found: ID does not exist" containerID="0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4" Nov 02 22:02:50.998988 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:50.998925 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4"} err="failed to get container status \"0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4\": rpc error: code = NotFound desc = could not find container \"0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4\": container with ID starting with 0ffb3151e71d395374d5827e40491373170e48973f14a57cb81db4c63ea82db4 not found: ID does not exist" Nov 02 22:02:51.068846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.068813 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b842\" (UniqueName: \"kubernetes.io/projected/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-kube-api-access-6b842\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.068846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.068836 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.068846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.068845 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdcaf39-4003-4ee2-9886-a1f46cc1857a-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.116682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.116649 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh"] Nov 02 22:02:51.122769 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.122739 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bcb857/test-model-bcb857-kserve-5d7f8b8989-bmbxh"] Nov 02 22:02:51.327200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.327162 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:51.656948 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.656871 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:02:51.662243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.662223 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:02:51.665231 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.665212 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:02:51.670465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.670448 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:02:51.676861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.676846 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:02:51.682548 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:51.682533 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:02:52.584038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:52.584007 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdcaf39-4003-4ee2-9886-a1f46cc1857a" path="/var/lib/kubelet/pods/2cdcaf39-4003-4ee2-9886-a1f46cc1857a/volumes" Nov 02 22:02:53.194102 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.194065 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2"] Nov 02 22:02:53.561536 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.561506 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:53.679685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.679641 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:53.698121 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.698083 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" podUID="d251ed63-5ecc-4824-b440-4c7d5f363ff3" containerName="payload-processing" containerID="cri-o://b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39" gracePeriod=25 Nov 02 22:02:53.800209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.800163 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" podUID="3fb20a5f-ae15-4d73-be8e-e56145047134" containerName="payload-pre-processing" containerID="cri-o://503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a" gracePeriod=25 Nov 02 22:02:53.924592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.924553 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:02:53.925775 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.925750 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:02:53.976037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:53.976008 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 22:02:54.013377 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.013346 4441 generic.go:356] "Generic (PLEG): container finished" podID="d251ed63-5ecc-4824-b440-4c7d5f363ff3" containerID="b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39" exitCode=0 Nov 02 22:02:54.013550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.013389 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" Nov 02 22:02:54.013550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.013430 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" event={"ID":"d251ed63-5ecc-4824-b440-4c7d5f363ff3","Type":"ContainerDied","Data":"b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39"} Nov 02 22:02:54.013550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.013484 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf" event={"ID":"d251ed63-5ecc-4824-b440-4c7d5f363ff3","Type":"ContainerDied","Data":"5d611b64adb2b50da0e6fd7f93562078ac9638eac2a986e6caa08ded98bea386"} Nov 02 22:02:54.013550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.013514 4441 scope.go:122] "RemoveContainer" containerID="b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39" Nov 02 22:02:54.024015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.023996 4441 scope.go:122] "RemoveContainer" containerID="b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39" Nov 02 22:02:54.024290 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:02:54.024271 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39\": container with ID starting with b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39 not found: ID does not exist" containerID="b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39" Nov 02 22:02:54.024327 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.024299 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39"} err="failed to get container status \"b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39\": rpc error: code = NotFound desc = could not find container \"b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39\": container with ID starting with b0f966d3d87a56522ce92826098f27c9788bd106936d5e6eb3102accd29b0f39 not found: ID does not exist" Nov 02 22:02:54.101904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.101813 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d251ed63-5ecc-4824-b440-4c7d5f363ff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d251ed63-5ecc-4824-b440-4c7d5f363ff3-plugins-config-volume\") pod \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " Nov 02 22:02:54.102088 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.101949 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d251ed63-5ecc-4824-b440-4c7d5f363ff3-kube-api-access-glrdd\" (UniqueName: \"kubernetes.io/projected/d251ed63-5ecc-4824-b440-4c7d5f363ff3-kube-api-access-glrdd\") pod \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\" (UID: \"d251ed63-5ecc-4824-b440-4c7d5f363ff3\") " Nov 02 22:02:54.102250 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.102229 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d251ed63-5ecc-4824-b440-4c7d5f363ff3-plugins-config-volume" pod "d251ed63-5ecc-4824-b440-4c7d5f363ff3" (UID: "d251ed63-5ecc-4824-b440-4c7d5f363ff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:54.102463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.102446 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d251ed63-5ecc-4824-b440-4c7d5f363ff3-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:54.104071 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.104044 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d251ed63-5ecc-4824-b440-4c7d5f363ff3-kube-api-access-glrdd" pod "d251ed63-5ecc-4824-b440-4c7d5f363ff3" (UID: "d251ed63-5ecc-4824-b440-4c7d5f363ff3"). InnerVolumeSpecName "kube-api-access-glrdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:54.203587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.203555 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glrdd\" (UniqueName: \"kubernetes.io/projected/d251ed63-5ecc-4824-b440-4c7d5f363ff3-kube-api-access-glrdd\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:54.429257 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.429178 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf"] Nov 02 22:02:54.432741 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.432716 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3566f9-6d8fc84c79-l9lpf"] Nov 02 22:02:54.592375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:54.592340 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d251ed63-5ecc-4824-b440-4c7d5f363ff3" path="/var/lib/kubelet/pods/d251ed63-5ecc-4824-b440-4c7d5f363ff3/volumes" Nov 02 22:02:55.848887 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:55.848853 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:55.966649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:55.966616 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:56.083584 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:56.083547 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:57.393533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:57.393489 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:58.194994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:58.194922 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" podUID="ed5a550d-c276-4810-93e2-38b30676cbe3" containerName="payload-processing" containerID="cri-o://555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b" gracePeriod=25 Nov 02 22:02:59.732768 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:02:59.732716 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" podUID="3fb20a5f-ae15-4d73-be8e-e56145047134" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:03:00.507864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:00.507811 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" podUID="ed5a550d-c276-4810-93e2-38b30676cbe3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:03:00.574404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:00.574351 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:02.329915 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.329828 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" containerID="cri-o://25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836" gracePeriod=45 Nov 02 22:03:02.613306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.613275 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 22:03:02.723420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723388 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07e2fd5-f023-4507-aa4a-854a9b490a1d-kube-api-access-xl4k8\" (UniqueName: \"kubernetes.io/projected/d07e2fd5-f023-4507-aa4a-854a9b490a1d-kube-api-access-xl4k8\") pod \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " Nov 02 22:03:02.723614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723445 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-home\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-home\") pod \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " Nov 02 22:03:02.723614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723485 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tls-certs\") pod \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " Nov 02 22:03:02.723614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723602 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-model-cache\") pod \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " Nov 02 22:03:02.723795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723691 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-dshm\") pod \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " Nov 02 22:03:02.723795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723723 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tmp-dir\") pod \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\" (UID: \"d07e2fd5-f023-4507-aa4a-854a9b490a1d\") " Nov 02 22:03:02.723795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723746 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-home" pod "d07e2fd5-f023-4507-aa4a-854a9b490a1d" (UID: "d07e2fd5-f023-4507-aa4a-854a9b490a1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.724161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.723952 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-model-cache" pod "d07e2fd5-f023-4507-aa4a-854a9b490a1d" (UID: "d07e2fd5-f023-4507-aa4a-854a9b490a1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.724300 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.724221 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.724300 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.724244 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.724853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.724480 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tmp-dir" pod "d07e2fd5-f023-4507-aa4a-854a9b490a1d" (UID: "d07e2fd5-f023-4507-aa4a-854a9b490a1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.731437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.731399 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tls-certs" pod "d07e2fd5-f023-4507-aa4a-854a9b490a1d" (UID: "d07e2fd5-f023-4507-aa4a-854a9b490a1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:02.731437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.731421 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-dshm" pod "d07e2fd5-f023-4507-aa4a-854a9b490a1d" (UID: "d07e2fd5-f023-4507-aa4a-854a9b490a1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.731824 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.731803 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07e2fd5-f023-4507-aa4a-854a9b490a1d-kube-api-access-xl4k8" pod "d07e2fd5-f023-4507-aa4a-854a9b490a1d" (UID: "d07e2fd5-f023-4507-aa4a-854a9b490a1d"). InnerVolumeSpecName "kube-api-access-xl4k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:02.825388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.825354 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.825388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.825385 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.825599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.825400 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl4k8\" (UniqueName: \"kubernetes.io/projected/d07e2fd5-f023-4507-aa4a-854a9b490a1d-kube-api-access-xl4k8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.825599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:02.825410 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07e2fd5-f023-4507-aa4a-854a9b490a1d-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:03.087649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.087614 4441 generic.go:356] "Generic (PLEG): container finished" podID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerID="25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836" exitCode=0 Nov 02 22:03:03.087649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.087651 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" event={"ID":"d07e2fd5-f023-4507-aa4a-854a9b490a1d","Type":"ContainerDied","Data":"25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836"} Nov 02 22:03:03.087916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.087668 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" Nov 02 22:03:03.087916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.087687 4441 scope.go:122] "RemoveContainer" containerID="25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836" Nov 02 22:03:03.087916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.087678 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6" event={"ID":"d07e2fd5-f023-4507-aa4a-854a9b490a1d","Type":"ContainerDied","Data":"66d4bd36dd9ede51662f56105446b8291894bb175016ea10ca9f990a49f03a1e"} Nov 02 22:03:03.098404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.098386 4441 scope.go:122] "RemoveContainer" containerID="25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836" Nov 02 22:03:03.098701 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:03:03.098671 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836\": container with ID starting with 25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836 not found: ID does not exist" containerID="25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836" Nov 02 22:03:03.098757 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.098707 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836"} err="failed to get container status \"25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836\": rpc error: code = NotFound desc = could not find container \"25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836\": container with ID starting with 25d4559392e99b47058ea17a6ae0ab77d2924c618a6e1e6dd86767ccbcc2e836 not found: ID does not exist" Nov 02 22:03:03.244015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.243978 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6"] Nov 02 22:03:03.247525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:03.247500 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/simulator-w1-c2dd3b-kserve-67b97458d4-dngn6"] Nov 02 22:03:04.584377 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:04.584348 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" path="/var/lib/kubelet/pods/d07e2fd5-f023-4507-aa4a-854a9b490a1d/volumes" Nov 02 22:03:05.170856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:05.170822 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:09.099122 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:09.099085 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:09.732936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:09.732896 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" podUID="3fb20a5f-ae15-4d73-be8e-e56145047134" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:03:10.189073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:10.189041 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:10.507733 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:10.507643 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" podUID="ed5a550d-c276-4810-93e2-38b30676cbe3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:03:10.556680 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:10.556649 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:10.614302 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:03:10.614270 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:03:10.845320 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:10.845288 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:11.063686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:11.063654 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w"] Nov 02 22:03:11.108110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:11.108026 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3566f9-z6v7w"] Nov 02 22:03:11.446753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:11.446662 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:11.988285 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:11.988253 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:12.436415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:12.436373 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:12.585635 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:12.585601 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4c7c37-3ea7-4393-b6c0-79243a316c64" path="/var/lib/kubelet/pods/ba4c7c37-3ea7-4393-b6c0-79243a316c64/volumes" Nov 02 22:03:12.704521 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:12.704440 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:12.915341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:12.915311 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9"] Nov 02 22:03:12.961931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:12.961849 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f4a242-46qw9"] Nov 02 22:03:13.015045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:13.015016 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:13.288743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:13.288669 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:14.067037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.067011 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 22:03:14.148703 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.148675 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb20a5f-ae15-4d73-be8e-e56145047134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb20a5f-ae15-4d73-be8e-e56145047134-plugins-config-volume\") pod \"3fb20a5f-ae15-4d73-be8e-e56145047134\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " Nov 02 22:03:14.148879 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.148860 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb20a5f-ae15-4d73-be8e-e56145047134-kube-api-access-bfbj9\" (UniqueName: \"kubernetes.io/projected/3fb20a5f-ae15-4d73-be8e-e56145047134-kube-api-access-bfbj9\") pod \"3fb20a5f-ae15-4d73-be8e-e56145047134\" (UID: \"3fb20a5f-ae15-4d73-be8e-e56145047134\") " Nov 02 22:03:14.149132 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.149108 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb20a5f-ae15-4d73-be8e-e56145047134-plugins-config-volume" pod "3fb20a5f-ae15-4d73-be8e-e56145047134" (UID: "3fb20a5f-ae15-4d73-be8e-e56145047134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:14.149293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.149278 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb20a5f-ae15-4d73-be8e-e56145047134-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:14.150906 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.150880 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb20a5f-ae15-4d73-be8e-e56145047134-kube-api-access-bfbj9" pod "3fb20a5f-ae15-4d73-be8e-e56145047134" (UID: "3fb20a5f-ae15-4d73-be8e-e56145047134"). InnerVolumeSpecName "kube-api-access-bfbj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:14.168387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.168364 4441 generic.go:356] "Generic (PLEG): container finished" podID="3fb20a5f-ae15-4d73-be8e-e56145047134" containerID="503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a" exitCode=0 Nov 02 22:03:14.168500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.168410 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" Nov 02 22:03:14.168500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.168448 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" event={"ID":"3fb20a5f-ae15-4d73-be8e-e56145047134","Type":"ContainerDied","Data":"503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a"} Nov 02 22:03:14.168585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.168500 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp" event={"ID":"3fb20a5f-ae15-4d73-be8e-e56145047134","Type":"ContainerDied","Data":"b0f85d578291a22bb8b24e3232c43124ef0646b677f13fe607c9975307a6b68b"} Nov 02 22:03:14.168585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.168523 4441 scope.go:122] "RemoveContainer" containerID="503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a" Nov 02 22:03:14.179720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.179699 4441 scope.go:122] "RemoveContainer" containerID="503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a" Nov 02 22:03:14.180121 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:03:14.180090 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a\": container with ID starting with 503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a not found: ID does not exist" containerID="503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a" Nov 02 22:03:14.180220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.180133 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a"} err="failed to get container status \"503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a\": rpc error: code = NotFound desc = could not find container \"503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a\": container with ID starting with 503b0f27a267e1f8b21d3327524797cdf33d38c6ceca2775622ed562bdf2047a not found: ID does not exist" Nov 02 22:03:14.250601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.250562 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfbj9\" (UniqueName: \"kubernetes.io/projected/3fb20a5f-ae15-4d73-be8e-e56145047134-kube-api-access-bfbj9\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:14.334314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.334225 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp"] Nov 02 22:03:14.338329 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.338303 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3566f9-5654cbf979-7v5tp"] Nov 02 22:03:14.584500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.584429 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb20a5f-ae15-4d73-be8e-e56145047134" path="/var/lib/kubelet/pods/3fb20a5f-ae15-4d73-be8e-e56145047134/volumes" Nov 02 22:03:14.584875 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:14.584861 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f9d8e0-40e5-4515-b2ce-ac258624223f" path="/var/lib/kubelet/pods/a6f9d8e0-40e5-4515-b2ce-ac258624223f/volumes" Nov 02 22:03:17.007513 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:17.007476 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:17.133776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:17.133731 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:17.353218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:17.353182 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:17.635471 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:17.635386 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:17.757829 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:17.757796 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:17.942303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:17.942228 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:18.460369 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.460312 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 22:03:18.504299 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.504252 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5a550d-c276-4810-93e2-38b30676cbe3-kube-api-access-stqnq\" (UniqueName: \"kubernetes.io/projected/ed5a550d-c276-4810-93e2-38b30676cbe3-kube-api-access-stqnq\") pod \"ed5a550d-c276-4810-93e2-38b30676cbe3\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " Nov 02 22:03:18.504511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.504313 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5a550d-c276-4810-93e2-38b30676cbe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a550d-c276-4810-93e2-38b30676cbe3-plugins-config-volume\") pod \"ed5a550d-c276-4810-93e2-38b30676cbe3\" (UID: \"ed5a550d-c276-4810-93e2-38b30676cbe3\") " Nov 02 22:03:18.504785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.504756 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5a550d-c276-4810-93e2-38b30676cbe3-plugins-config-volume" pod "ed5a550d-c276-4810-93e2-38b30676cbe3" (UID: "ed5a550d-c276-4810-93e2-38b30676cbe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:18.507418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.507390 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5a550d-c276-4810-93e2-38b30676cbe3-kube-api-access-stqnq" pod "ed5a550d-c276-4810-93e2-38b30676cbe3" (UID: "ed5a550d-c276-4810-93e2-38b30676cbe3"). InnerVolumeSpecName "kube-api-access-stqnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:18.605628 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.605586 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqnq\" (UniqueName: \"kubernetes.io/projected/ed5a550d-c276-4810-93e2-38b30676cbe3-kube-api-access-stqnq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:18.605628 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.605628 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a550d-c276-4810-93e2-38b30676cbe3-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:18.697377 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697334 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x"] Nov 02 22:03:18.697841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697826 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d251ed63-5ecc-4824-b440-4c7d5f363ff3" containerName="payload-processing" Nov 02 22:03:18.697841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697841 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d251ed63-5ecc-4824-b440-4c7d5f363ff3" containerName="payload-processing" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697859 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697864 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697879 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ed5a550d-c276-4810-93e2-38b30676cbe3" containerName="payload-processing" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697885 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5a550d-c276-4810-93e2-38b30676cbe3" containerName="payload-processing" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697893 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697898 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697904 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3b302d04-e3f2-4c17-8f08-e1939d74f8e0" containerName="revoke-keys" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697908 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b302d04-e3f2-4c17-8f08-e1939d74f8e0" containerName="revoke-keys" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697916 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3fb20a5f-ae15-4d73-be8e-e56145047134" containerName="payload-pre-processing" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697921 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb20a5f-ae15-4d73-be8e-e56145047134" containerName="payload-pre-processing" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697931 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697936 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697951 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2cdcaf39-4003-4ee2-9886-a1f46cc1857a" containerName="main" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697955 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdcaf39-4003-4ee2-9886-a1f46cc1857a" containerName="main" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697961 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f084e9bd-2791-43cb-b8df-dfd56dc51b9e" containerName="revoke-keys" Nov 02 22:03:18.698018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.697983 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f084e9bd-2791-43cb-b8df-dfd56dc51b9e" containerName="revoke-keys" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698070 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07e2fd5-f023-4507-aa4a-854a9b490a1d" containerName="main" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698083 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdcaf39-4003-4ee2-9886-a1f46cc1857a" containerName="main" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698090 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f084e9bd-2791-43cb-b8df-dfd56dc51b9e" containerName="revoke-keys" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698097 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d251ed63-5ecc-4824-b440-4c7d5f363ff3" containerName="payload-processing" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698105 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698115 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5a550d-c276-4810-93e2-38b30676cbe3" containerName="payload-processing" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698120 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e865ee9-3056-48a2-bed0-19b609cb1795" containerName="istio-proxy" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698127 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b302d04-e3f2-4c17-8f08-e1939d74f8e0" containerName="revoke-keys" Nov 02 22:03:18.698860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.698132 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb20a5f-ae15-4d73-be8e-e56145047134" containerName="payload-pre-processing" Nov 02 22:03:18.701609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.701589 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.751414 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.751367 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:03:18.751606 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.751517 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:03:18.751606 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.751582 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:03:18.751746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.751670 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:03:18.753929 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.753901 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x"] Nov 02 22:03:18.807629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.807585 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.807832 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.807715 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.807832 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.807756 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnl58\" (UniqueName: \"kubernetes.io/projected/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-kube-api-access-rnl58\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.909405 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.909371 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.909616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.909431 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.909616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.909452 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnl58\" (UniqueName: \"kubernetes.io/projected/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-kube-api-access-rnl58\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.910166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.910135 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.912297 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.912273 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:18.963553 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:18.963497 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnl58\" (UniqueName: \"kubernetes.io/projected/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-kube-api-access-rnl58\") pod \"maas-api-revoke-keys-e2e-inf-b-bcb857-t264x\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:19.018948 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.018923 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:19.134639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.134608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:19.217869 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.217779 4441 generic.go:356] "Generic (PLEG): container finished" podID="ed5a550d-c276-4810-93e2-38b30676cbe3" containerID="555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b" exitCode=0 Nov 02 22:03:19.217869 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.217851 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" Nov 02 22:03:19.218149 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.218029 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" event={"ID":"ed5a550d-c276-4810-93e2-38b30676cbe3","Type":"ContainerDied","Data":"555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b"} Nov 02 22:03:19.218149 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.218083 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2" event={"ID":"ed5a550d-c276-4810-93e2-38b30676cbe3","Type":"ContainerDied","Data":"39ca8f0e8d9fdb71554ad33ecfdb48516643179c9d1d9de53120fb9c2c6918f9"} Nov 02 22:03:19.218149 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.218116 4441 scope.go:122] "RemoveContainer" containerID="555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b" Nov 02 22:03:19.228603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.228584 4441 scope.go:122] "RemoveContainer" containerID="555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b" Nov 02 22:03:19.228903 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:03:19.228877 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b\": container with ID starting with 555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b not found: ID does not exist" containerID="555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b" Nov 02 22:03:19.228956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.228911 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b"} err="failed to get container status \"555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b\": rpc error: code = NotFound desc = could not find container \"555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b\": container with ID starting with 555a652aa54a2de53e461d630a9d80f7c2d60e718100e9b8afcd3d6f99dfe70b not found: ID does not exist" Nov 02 22:03:19.260869 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.260845 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:19.376421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.376337 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2"] Nov 02 22:03:19.381782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.381754 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c2dd3b-65f5c79bd5-z22x2"] Nov 02 22:03:19.424301 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.424233 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x"] Nov 02 22:03:19.438112 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:03:19.438078 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459bc042_b104_4d52_9f0b_e6d2d0bcc7e2.slice/crio-840aff0cd5aa63389c5eac72d3dc46e2d920f5819702055a5544cdd842ebd51d WatchSource:0}: Error finding container 840aff0cd5aa63389c5eac72d3dc46e2d920f5819702055a5544cdd842ebd51d: Status 404 returned error can't find the container with id 840aff0cd5aa63389c5eac72d3dc46e2d920f5819702055a5544cdd842ebd51d Nov 02 22:03:19.448863 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:19.448838 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:20.229154 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.229115 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" event={"ID":"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2","Type":"ContainerStarted","Data":"1d5072f94f5346fbb585d8c6841e8764b8baa82e9879304b2e7ba7ca7fd4a92c"} Nov 02 22:03:20.229653 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.229176 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" event={"ID":"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2","Type":"ContainerStarted","Data":"840aff0cd5aa63389c5eac72d3dc46e2d920f5819702055a5544cdd842ebd51d"} Nov 02 22:03:20.230857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.230835 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh"] Nov 02 22:03:20.234702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.234687 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.288249 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.288220 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh"] Nov 02 22:03:20.326401 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.326375 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.326544 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.326505 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzmq\" (UniqueName: \"kubernetes.io/projected/2e23bd79-ecbc-46ba-9173-246993fddf1b-kube-api-access-6bzmq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.326612 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.326597 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.337899 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.337856 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" podStartSLOduration=2.337843436 podStartE2EDuration="2.337843436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:20.336009574 +0000 UTC m=+3029.891637913" watchObservedRunningTime="2026-11-02 22:03:20.337843436 +0000 UTC m=+3029.893471773" Nov 02 22:03:20.427963 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.427925 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.428157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.428037 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzmq\" (UniqueName: \"kubernetes.io/projected/2e23bd79-ecbc-46ba-9173-246993fddf1b-kube-api-access-6bzmq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.428157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.428140 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.428752 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.428728 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.430235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.430213 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.482326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.482252 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzmq\" (UniqueName: \"kubernetes.io/projected/2e23bd79-ecbc-46ba-9173-246993fddf1b-kube-api-access-6bzmq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.547327 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.547304 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:20.586381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.586354 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5a550d-c276-4810-93e2-38b30676cbe3" path="/var/lib/kubelet/pods/ed5a550d-c276-4810-93e2-38b30676cbe3/volumes" Nov 02 22:03:20.917383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:20.917354 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh"] Nov 02 22:03:20.919010 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:03:20.918959 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e23bd79_ecbc_46ba_9173_246993fddf1b.slice/crio-fce91f0f598749579c0ccc0b240e69522df51d60c7d63df309684660e8b08ef8 WatchSource:0}: Error finding container fce91f0f598749579c0ccc0b240e69522df51d60c7d63df309684660e8b08ef8: Status 404 returned error can't find the container with id fce91f0f598749579c0ccc0b240e69522df51d60c7d63df309684660e8b08ef8 Nov 02 22:03:21.238627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:21.238545 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" event={"ID":"2e23bd79-ecbc-46ba-9173-246993fddf1b","Type":"ContainerStarted","Data":"fce91f0f598749579c0ccc0b240e69522df51d60c7d63df309684660e8b08ef8"} Nov 02 22:03:21.240316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:21.240283 4441 generic.go:356] "Generic (PLEG): container finished" podID="459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" containerID="1d5072f94f5346fbb585d8c6841e8764b8baa82e9879304b2e7ba7ca7fd4a92c" exitCode=0 Nov 02 22:03:21.240455 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:21.240344 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" event={"ID":"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2","Type":"ContainerDied","Data":"1d5072f94f5346fbb585d8c6841e8764b8baa82e9879304b2e7ba7ca7fd4a92c"} Nov 02 22:03:21.931177 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:21.931145 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:22.249091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.248997 4441 generic.go:356] "Generic (PLEG): container finished" podID="2e23bd79-ecbc-46ba-9173-246993fddf1b" containerID="b5630e85a024da68aaf1bdebcfeba83fe3ab5bbf2dd799d96fea6ffc55b8af82" exitCode=0 Nov 02 22:03:22.249091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.249055 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" event={"ID":"2e23bd79-ecbc-46ba-9173-246993fddf1b","Type":"ContainerDied","Data":"b5630e85a024da68aaf1bdebcfeba83fe3ab5bbf2dd799d96fea6ffc55b8af82"} Nov 02 22:03:22.404593 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.404559 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:22.448241 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.448217 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-kube-api-access-rnl58\" (UniqueName: \"kubernetes.io/projected/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-kube-api-access-rnl58\") pod \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " Nov 02 22:03:22.448392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.448277 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-openshift-ca\") pod \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " Nov 02 22:03:22.448465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.448396 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-xks-ca\") pod \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\" (UID: \"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2\") " Nov 02 22:03:22.448729 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.448701 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-openshift-ca" pod "459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" (UID: "459bc042-b104-4d52-9f0b-e6d2d0bcc7e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:22.450368 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.450344 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-kube-api-access-rnl58" pod "459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" (UID: "459bc042-b104-4d52-9f0b-e6d2d0bcc7e2"). InnerVolumeSpecName "kube-api-access-rnl58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:22.450441 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.450396 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-xks-ca" pod "459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" (UID: "459bc042-b104-4d52-9f0b-e6d2d0bcc7e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:22.549470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.549400 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:22.549470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.549427 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnl58\" (UniqueName: \"kubernetes.io/projected/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-kube-api-access-rnl58\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:22.549470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:22.549437 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:23.192275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.192239 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:23.256754 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.256726 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" Nov 02 22:03:23.257146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.256769 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x" event={"ID":"459bc042-b104-4d52-9f0b-e6d2d0bcc7e2","Type":"ContainerDied","Data":"840aff0cd5aa63389c5eac72d3dc46e2d920f5819702055a5544cdd842ebd51d"} Nov 02 22:03:23.257146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.256814 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840aff0cd5aa63389c5eac72d3dc46e2d920f5819702055a5544cdd842ebd51d" Nov 02 22:03:23.401879 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.401855 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:23.456421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.456346 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e23bd79-ecbc-46ba-9173-246993fddf1b-kube-api-access-6bzmq\" (UniqueName: \"kubernetes.io/projected/2e23bd79-ecbc-46ba-9173-246993fddf1b-kube-api-access-6bzmq\") pod \"2e23bd79-ecbc-46ba-9173-246993fddf1b\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " Nov 02 22:03:23.456421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.456402 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-xks-ca\") pod \"2e23bd79-ecbc-46ba-9173-246993fddf1b\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " Nov 02 22:03:23.456645 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.456449 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-openshift-ca\") pod \"2e23bd79-ecbc-46ba-9173-246993fddf1b\" (UID: \"2e23bd79-ecbc-46ba-9173-246993fddf1b\") " Nov 02 22:03:23.456885 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.456864 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-openshift-ca" pod "2e23bd79-ecbc-46ba-9173-246993fddf1b" (UID: "2e23bd79-ecbc-46ba-9173-246993fddf1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:23.458511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.458486 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-xks-ca" pod "2e23bd79-ecbc-46ba-9173-246993fddf1b" (UID: "2e23bd79-ecbc-46ba-9173-246993fddf1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:23.458643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.458580 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e23bd79-ecbc-46ba-9173-246993fddf1b-kube-api-access-6bzmq" pod "2e23bd79-ecbc-46ba-9173-246993fddf1b" (UID: "2e23bd79-ecbc-46ba-9173-246993fddf1b"). InnerVolumeSpecName "kube-api-access-6bzmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:23.557513 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.557483 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bzmq\" (UniqueName: \"kubernetes.io/projected/2e23bd79-ecbc-46ba-9173-246993fddf1b-kube-api-access-6bzmq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:23.557513 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.557509 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:23.557723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:23.557521 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e23bd79-ecbc-46ba-9173-246993fddf1b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:24.265645 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:24.265608 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" Nov 02 22:03:24.266082 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:24.265649 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh" event={"ID":"2e23bd79-ecbc-46ba-9173-246993fddf1b","Type":"ContainerDied","Data":"fce91f0f598749579c0ccc0b240e69522df51d60c7d63df309684660e8b08ef8"} Nov 02 22:03:24.266082 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:24.265695 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fce91f0f598749579c0ccc0b240e69522df51d60c7d63df309684660e8b08ef8" Nov 02 22:03:26.822174 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:26.822142 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:28.696444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:28.696411 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:30.199828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:30.199788 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz"] Nov 02 22:03:30.430926 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:30.430894 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:30.553797 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:30.553720 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:31.320393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:31.320354 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:31.473359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:31.473321 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:31.594661 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:31.594569 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:33.044534 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:33.044498 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4"] Nov 02 22:03:33.091316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:33.091289 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410f0750-zr5m4"] Nov 02 22:03:34.585354 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:34.585210 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1180da75-b7a5-4a10-a580-430af8675ac4" path="/var/lib/kubelet/pods/1180da75-b7a5-4a10-a580-430af8675ac4/volumes" Nov 02 22:03:34.970683 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:34.970600 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:35.087931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.087901 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:35.200981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.200926 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" podUID="b3f7af50-2c16-4253-9666-9b20fcc311b7" containerName="payload-pre-processing" containerID="cri-o://f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410" gracePeriod=25 Nov 02 22:03:35.541456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.541422 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x"] Nov 02 22:03:35.541947 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.541931 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2e23bd79-ecbc-46ba-9173-246993fddf1b" containerName="revoke-keys" Nov 02 22:03:35.542015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.541949 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e23bd79-ecbc-46ba-9173-246993fddf1b" containerName="revoke-keys" Nov 02 22:03:35.542015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.541959 4441 cpu_manager.go:414] "containerMap: removing container" podUID="459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" containerName="revoke-keys" Nov 02 22:03:35.542015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.541976 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" containerName="revoke-keys" Nov 02 22:03:35.542120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.542085 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e23bd79-ecbc-46ba-9173-246993fddf1b" containerName="revoke-keys" Nov 02 22:03:35.542120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.542092 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" containerName="revoke-keys" Nov 02 22:03:35.546652 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.546635 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.592630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.592599 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr"] Nov 02 22:03:35.598681 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.598659 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x"] Nov 02 22:03:35.598807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.598746 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.636096 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.636069 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:03:35.636096 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.636076 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:03:35.639580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.639550 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:03:35.639701 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.639601 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:03:35.651492 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.651469 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr"] Nov 02 22:03:35.692720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.692692 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8v5\" (UniqueName: \"kubernetes.io/projected/72e26b62-b3da-40d2-b692-6241fd2f0b62-kube-api-access-tj8v5\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.692930 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.692910 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.693098 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.693072 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ttd\" (UniqueName: \"kubernetes.io/projected/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-kube-api-access-l2ttd\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.693242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.693114 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.693242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.693209 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.693390 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.693301 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.729008 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.728959 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:35.794739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.794672 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8v5\" (UniqueName: \"kubernetes.io/projected/72e26b62-b3da-40d2-b692-6241fd2f0b62-kube-api-access-tj8v5\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.794739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.794708 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.794739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.794730 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ttd\" (UniqueName: \"kubernetes.io/projected/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-kube-api-access-l2ttd\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.795044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.794752 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.795044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.794771 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.795044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.794808 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.795515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.795491 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.795617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.795599 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.797217 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.797190 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.797317 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.797201 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.854823 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.854786 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8v5\" (UniqueName: \"kubernetes.io/projected/72e26b62-b3da-40d2-b692-6241fd2f0b62-kube-api-access-tj8v5\") pod \"maas-api-revoke-sub-38bc185c-6zt4x\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.855678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.855613 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ttd\" (UniqueName: \"kubernetes.io/projected/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-kube-api-access-l2ttd\") pod \"maas-api-revoke-sub-06073ecc-jgwsr\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.860495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.860469 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:35.914312 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.914248 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:35.966383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:35.966143 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" podUID="b3f7af50-2c16-4253-9666-9b20fcc311b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:03:36.228858 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:03:36.228808 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e26b62_b3da_40d2_b692_6241fd2f0b62.slice/crio-6a1f0bd1c979f820ab075a57a598eeffd66923fd3e2b7ad37fd655f1653f5b2d WatchSource:0}: Error finding container 6a1f0bd1c979f820ab075a57a598eeffd66923fd3e2b7ad37fd655f1653f5b2d: Status 404 returned error can't find the container with id 6a1f0bd1c979f820ab075a57a598eeffd66923fd3e2b7ad37fd655f1653f5b2d Nov 02 22:03:36.229301 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:36.229277 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x"] Nov 02 22:03:36.328789 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:36.328758 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr"] Nov 02 22:03:36.329026 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:03:36.329006 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f4cf9c_ab75_4f17_a82a_f125cfc4489b.slice/crio-1d433049b24fd881c3a560013960cb65d959dca1e6d60b108764a08cafa8d086 WatchSource:0}: Error finding container 1d433049b24fd881c3a560013960cb65d959dca1e6d60b108764a08cafa8d086: Status 404 returned error can't find the container with id 1d433049b24fd881c3a560013960cb65d959dca1e6d60b108764a08cafa8d086 Nov 02 22:03:36.358621 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:36.358597 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" event={"ID":"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b","Type":"ContainerStarted","Data":"1d433049b24fd881c3a560013960cb65d959dca1e6d60b108764a08cafa8d086"} Nov 02 22:03:36.359939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:36.359918 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" event={"ID":"72e26b62-b3da-40d2-b692-6241fd2f0b62","Type":"ContainerStarted","Data":"6a1f0bd1c979f820ab075a57a598eeffd66923fd3e2b7ad37fd655f1653f5b2d"} Nov 02 22:03:37.192982 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.192941 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:37.370413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.370326 4441 generic.go:356] "Generic (PLEG): container finished" podID="72e26b62-b3da-40d2-b692-6241fd2f0b62" containerID="a71b7d1785c818e5e2805e252bb1446d5c60c2c395fe5778904f4c1517ccf484" exitCode=0 Nov 02 22:03:37.370592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.370418 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" event={"ID":"72e26b62-b3da-40d2-b692-6241fd2f0b62","Type":"ContainerDied","Data":"a71b7d1785c818e5e2805e252bb1446d5c60c2c395fe5778904f4c1517ccf484"} Nov 02 22:03:37.372173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.372150 4441 generic.go:356] "Generic (PLEG): container finished" podID="a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" containerID="fe92ccb99ab16a0f40b9dcf5d06c5144da609e66a3e56898aec8cdcbc525bdd6" exitCode=0 Nov 02 22:03:37.372285 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.372184 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" event={"ID":"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b","Type":"ContainerDied","Data":"fe92ccb99ab16a0f40b9dcf5d06c5144da609e66a3e56898aec8cdcbc525bdd6"} Nov 02 22:03:37.401036 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.401012 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:37.527506 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.527464 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:37.676183 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.676091 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x"] Nov 02 22:03:37.679578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:37.679553 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1243c-msb2x"] Nov 02 22:03:38.055030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.054925 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:38.545874 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.545852 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:38.571172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.571144 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:38.584067 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.584040 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f084e9bd-2791-43cb-b8df-dfd56dc51b9e" path="/var/lib/kubelet/pods/f084e9bd-2791-43cb-b8df-dfd56dc51b9e/volumes" Nov 02 22:03:38.628678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.628647 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-xks-ca\") pod \"72e26b62-b3da-40d2-b692-6241fd2f0b62\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " Nov 02 22:03:38.628808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.628704 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e26b62-b3da-40d2-b692-6241fd2f0b62-kube-api-access-tj8v5\" (UniqueName: \"kubernetes.io/projected/72e26b62-b3da-40d2-b692-6241fd2f0b62-kube-api-access-tj8v5\") pod \"72e26b62-b3da-40d2-b692-6241fd2f0b62\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " Nov 02 22:03:38.628808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.628799 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-openshift-ca\") pod \"72e26b62-b3da-40d2-b692-6241fd2f0b62\" (UID: \"72e26b62-b3da-40d2-b692-6241fd2f0b62\") " Nov 02 22:03:38.629175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.629152 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-openshift-ca" pod "72e26b62-b3da-40d2-b692-6241fd2f0b62" (UID: "72e26b62-b3da-40d2-b692-6241fd2f0b62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:38.630822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.630801 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-xks-ca" pod "72e26b62-b3da-40d2-b692-6241fd2f0b62" (UID: "72e26b62-b3da-40d2-b692-6241fd2f0b62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:38.630920 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.630804 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e26b62-b3da-40d2-b692-6241fd2f0b62-kube-api-access-tj8v5" pod "72e26b62-b3da-40d2-b692-6241fd2f0b62" (UID: "72e26b62-b3da-40d2-b692-6241fd2f0b62"). InnerVolumeSpecName "kube-api-access-tj8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:38.729473 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.729443 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-openshift-ca\") pod \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " Nov 02 22:03:38.729672 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.729521 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-kube-api-access-l2ttd\" (UniqueName: \"kubernetes.io/projected/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-kube-api-access-l2ttd\") pod \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " Nov 02 22:03:38.729672 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.729581 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-xks-ca\") pod \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\" (UID: \"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b\") " Nov 02 22:03:38.730103 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.729855 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-openshift-ca" pod "a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" (UID: "a4f4cf9c-ab75-4f17-a82a-f125cfc4489b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:38.730364 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.730342 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8v5\" (UniqueName: \"kubernetes.io/projected/72e26b62-b3da-40d2-b692-6241fd2f0b62-kube-api-access-tj8v5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:38.730443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.730370 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:38.730443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.730386 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:38.730443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.730404 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e26b62-b3da-40d2-b692-6241fd2f0b62-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:38.731755 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.731736 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-xks-ca" pod "a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" (UID: "a4f4cf9c-ab75-4f17-a82a-f125cfc4489b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:38.731899 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.731883 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-kube-api-access-l2ttd" pod "a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" (UID: "a4f4cf9c-ab75-4f17-a82a-f125cfc4489b"). InnerVolumeSpecName "kube-api-access-l2ttd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:38.831288 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.831260 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:38.831288 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:38.831286 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ttd\" (UniqueName: \"kubernetes.io/projected/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b-kube-api-access-l2ttd\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:39.391254 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:39.391224 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" Nov 02 22:03:39.391488 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:39.391260 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x" event={"ID":"72e26b62-b3da-40d2-b692-6241fd2f0b62","Type":"ContainerDied","Data":"6a1f0bd1c979f820ab075a57a598eeffd66923fd3e2b7ad37fd655f1653f5b2d"} Nov 02 22:03:39.391488 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:39.391312 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a1f0bd1c979f820ab075a57a598eeffd66923fd3e2b7ad37fd655f1653f5b2d" Nov 02 22:03:39.392655 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:39.392631 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" Nov 02 22:03:39.392782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:39.392675 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr" event={"ID":"a4f4cf9c-ab75-4f17-a82a-f125cfc4489b","Type":"ContainerDied","Data":"1d433049b24fd881c3a560013960cb65d959dca1e6d60b108764a08cafa8d086"} Nov 02 22:03:39.392782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:39.392718 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d433049b24fd881c3a560013960cb65d959dca1e6d60b108764a08cafa8d086" Nov 02 22:03:40.615878 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:03:40.615835 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:03:42.272016 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:42.271982 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:43.118491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:43.118454 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:43.238835 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:43.238802 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:43.951785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:43.951753 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:44.075675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:44.075640 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:44.293394 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:44.293304 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:45.075622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.075581 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld"] Nov 02 22:03:45.076339 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.076317 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" containerName="revoke-keys" Nov 02 22:03:45.076444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.076341 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" containerName="revoke-keys" Nov 02 22:03:45.076444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.076380 4441 cpu_manager.go:414] "containerMap: removing container" podUID="72e26b62-b3da-40d2-b692-6241fd2f0b62" containerName="revoke-keys" Nov 02 22:03:45.076444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.076388 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e26b62-b3da-40d2-b692-6241fd2f0b62" containerName="revoke-keys" Nov 02 22:03:45.076615 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.076526 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" containerName="revoke-keys" Nov 02 22:03:45.076615 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.076545 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e26b62-b3da-40d2-b692-6241fd2f0b62" containerName="revoke-keys" Nov 02 22:03:45.081082 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.081061 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.130521 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.130494 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:03:45.130892 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.130867 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld"] Nov 02 22:03:45.173292 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.173259 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:03:45.173292 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.173288 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:03:45.173504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.173259 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:03:45.196738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.196699 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.196884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.196801 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gtkn\" (UniqueName: \"kubernetes.io/projected/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-kube-api-access-6gtkn\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.196936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.196917 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.298334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.298296 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gtkn\" (UniqueName: \"kubernetes.io/projected/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-kube-api-access-6gtkn\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.298487 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.298377 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.298546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.298491 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.299373 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.299344 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.300765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.300743 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.352199 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.352139 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gtkn\" (UniqueName: \"kubernetes.io/projected/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-kube-api-access-6gtkn\") pod \"maas-api-revoke-keys-e2e-inf-b-70d237-n5qld\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.397797 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.397763 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:45.768328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.768299 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld"] Nov 02 22:03:45.768664 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:03:45.768630 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabaa10f_0506_45c5_ab4f_4e9a10aa3216.slice/crio-1ee992a4185a2f7a59fa1c3090fa587610ee6c78a89881e1965da64689323a51 WatchSource:0}: Error finding container 1ee992a4185a2f7a59fa1c3090fa587610ee6c78a89881e1965da64689323a51: Status 404 returned error can't find the container with id 1ee992a4185a2f7a59fa1c3090fa587610ee6c78a89881e1965da64689323a51 Nov 02 22:03:45.966294 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:45.966254 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" podUID="b3f7af50-2c16-4253-9666-9b20fcc311b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:03:46.043118 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:46.043051 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q"] Nov 02 22:03:46.046533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:46.046511 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4810d2a3-hsh7q"] Nov 02 22:03:46.447109 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:46.447071 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" event={"ID":"cabaa10f-0506-45c5-ab4f-4e9a10aa3216","Type":"ContainerStarted","Data":"1fed1e1c5f0903b40e12e8b3d2aa867b4b2628677ae962c344dd367334932323"} Nov 02 22:03:46.447472 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:46.447125 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" event={"ID":"cabaa10f-0506-45c5-ab4f-4e9a10aa3216","Type":"ContainerStarted","Data":"1ee992a4185a2f7a59fa1c3090fa587610ee6c78a89881e1965da64689323a51"} Nov 02 22:03:46.585957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:46.585924 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814422d2-590b-49e4-8da2-d92333945435" path="/var/lib/kubelet/pods/814422d2-590b-49e4-8da2-d92333945435/volumes" Nov 02 22:03:46.965110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:46.965072 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:47.456045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:47.456009 4441 generic.go:356] "Generic (PLEG): container finished" podID="cabaa10f-0506-45c5-ab4f-4e9a10aa3216" containerID="1fed1e1c5f0903b40e12e8b3d2aa867b4b2628677ae962c344dd367334932323" exitCode=0 Nov 02 22:03:47.456432 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:47.456099 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" event={"ID":"cabaa10f-0506-45c5-ab4f-4e9a10aa3216","Type":"ContainerDied","Data":"1fed1e1c5f0903b40e12e8b3d2aa867b4b2628677ae962c344dd367334932323"} Nov 02 22:03:48.658244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.658212 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:48.746244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.746152 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-openshift-ca\") pod \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " Nov 02 22:03:48.746244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.746232 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-xks-ca\") pod \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " Nov 02 22:03:48.746499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.746333 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-kube-api-access-6gtkn\" (UniqueName: \"kubernetes.io/projected/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-kube-api-access-6gtkn\") pod \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\" (UID: \"cabaa10f-0506-45c5-ab4f-4e9a10aa3216\") " Nov 02 22:03:48.746602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.746579 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-openshift-ca" pod "cabaa10f-0506-45c5-ab4f-4e9a10aa3216" (UID: "cabaa10f-0506-45c5-ab4f-4e9a10aa3216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:48.746779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.746762 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:48.748377 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.748353 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-xks-ca" pod "cabaa10f-0506-45c5-ab4f-4e9a10aa3216" (UID: "cabaa10f-0506-45c5-ab4f-4e9a10aa3216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:48.748465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.748448 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-kube-api-access-6gtkn" pod "cabaa10f-0506-45c5-ab4f-4e9a10aa3216" (UID: "cabaa10f-0506-45c5-ab4f-4e9a10aa3216"). InnerVolumeSpecName "kube-api-access-6gtkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:48.848375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.848341 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gtkn\" (UniqueName: \"kubernetes.io/projected/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-kube-api-access-6gtkn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:48.848375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:48.848373 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabaa10f-0506-45c5-ab4f-4e9a10aa3216-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:49.496036 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:49.496005 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" Nov 02 22:03:49.496227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:49.496004 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld" event={"ID":"cabaa10f-0506-45c5-ab4f-4e9a10aa3216","Type":"ContainerDied","Data":"1ee992a4185a2f7a59fa1c3090fa587610ee6c78a89881e1965da64689323a51"} Nov 02 22:03:49.496227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:49.496142 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ee992a4185a2f7a59fa1c3090fa587610ee6c78a89881e1965da64689323a51" Nov 02 22:03:52.037745 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:52.037698 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:54.905954 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:54.905908 4441 scope.go:122] "RemoveContainer" containerID="082426cc3d649e77441503edc969ab98044f2baff9239cc2debb57a1961b12a1" Nov 02 22:03:55.487269 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.487241 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 22:03:55.541514 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.541484 4441 generic.go:356] "Generic (PLEG): container finished" podID="b3f7af50-2c16-4253-9666-9b20fcc311b7" containerID="f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410" exitCode=0 Nov 02 22:03:55.541675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.541525 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" Nov 02 22:03:55.541675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.541582 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" event={"ID":"b3f7af50-2c16-4253-9666-9b20fcc311b7","Type":"ContainerDied","Data":"f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410"} Nov 02 22:03:55.541675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.541645 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz" event={"ID":"b3f7af50-2c16-4253-9666-9b20fcc311b7","Type":"ContainerDied","Data":"a68280d3fb5356f68032c946bfd94d6ab42bbed6d7fe08861c7014e5943c669a"} Nov 02 22:03:55.541800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.541683 4441 scope.go:122] "RemoveContainer" containerID="f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410" Nov 02 22:03:55.542640 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.542613 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f7af50-2c16-4253-9666-9b20fcc311b7-kube-api-access-skxjb\" (UniqueName: \"kubernetes.io/projected/b3f7af50-2c16-4253-9666-9b20fcc311b7-kube-api-access-skxjb\") pod \"b3f7af50-2c16-4253-9666-9b20fcc311b7\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " Nov 02 22:03:55.542770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.542662 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f7af50-2c16-4253-9666-9b20fcc311b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f7af50-2c16-4253-9666-9b20fcc311b7-plugins-config-volume\") pod \"b3f7af50-2c16-4253-9666-9b20fcc311b7\" (UID: \"b3f7af50-2c16-4253-9666-9b20fcc311b7\") " Nov 02 22:03:55.543129 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.543107 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f7af50-2c16-4253-9666-9b20fcc311b7-plugins-config-volume" pod "b3f7af50-2c16-4253-9666-9b20fcc311b7" (UID: "b3f7af50-2c16-4253-9666-9b20fcc311b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:55.544808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.544779 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f7af50-2c16-4253-9666-9b20fcc311b7-kube-api-access-skxjb" pod "b3f7af50-2c16-4253-9666-9b20fcc311b7" (UID: "b3f7af50-2c16-4253-9666-9b20fcc311b7"). InnerVolumeSpecName "kube-api-access-skxjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:55.559023 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.558958 4441 scope.go:122] "RemoveContainer" containerID="f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410" Nov 02 22:03:55.559271 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:03:55.559254 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410\": container with ID starting with f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410 not found: ID does not exist" containerID="f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410" Nov 02 22:03:55.559358 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.559278 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410"} err="failed to get container status \"f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410\": rpc error: code = NotFound desc = could not find container \"f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410\": container with ID starting with f0f74d9ed1f8db32741f913cb011c4c305729f7103464c968391700ff673f410 not found: ID does not exist" Nov 02 22:03:55.643568 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.643537 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxjb\" (UniqueName: \"kubernetes.io/projected/b3f7af50-2c16-4253-9666-9b20fcc311b7-kube-api-access-skxjb\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:55.643568 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.643566 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3f7af50-2c16-4253-9666-9b20fcc311b7-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:55.957653 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:55.957618 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz"] Nov 02 22:03:56.007043 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:56.007009 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcb857-55c67bc8f9-44fjz"] Nov 02 22:03:56.495419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:56.495377 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72"] Nov 02 22:03:56.586523 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:56.586489 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f7af50-2c16-4253-9666-9b20fcc311b7" path="/var/lib/kubelet/pods/b3f7af50-2c16-4253-9666-9b20fcc311b7/volumes" Nov 02 22:03:56.600251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:56.599163 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg"] Nov 02 22:03:56.826037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:56.825996 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:56.941932 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:56.941899 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:57.560963 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:57.560927 4441 generic.go:356] "Generic (PLEG): container finished" podID="801b2255-648f-4abe-91f6-d29770490815" containerID="4c00219cd4fa9c1647445ceb190b25981255c0b09d2ba0a319eb7cb6df54677f" exitCode=0 Nov 02 22:03:57.561519 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:57.561017 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" event={"ID":"801b2255-648f-4abe-91f6-d29770490815","Type":"ContainerDied","Data":"4c00219cd4fa9c1647445ceb190b25981255c0b09d2ba0a319eb7cb6df54677f"} Nov 02 22:03:57.561613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:57.561555 4441 scope.go:122] "RemoveContainer" containerID="4c00219cd4fa9c1647445ceb190b25981255c0b09d2ba0a319eb7cb6df54677f" Nov 02 22:03:57.563003 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:57.562960 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:03:57.563075 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:57.563044 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:03:57.563113 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:57.563078 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:03:57.730012 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:57.729959 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:58.575992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:58.574264 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" event={"ID":"801b2255-648f-4abe-91f6-d29770490815","Type":"ContainerStarted","Data":"57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42"} Nov 02 22:03:59.217658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:59.217617 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:03:59.218605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:59.218573 4441 patch_prober.go:29] interesting pod/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 02 22:03:59.218746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:03:59.218623 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 02 22:04:00.217428 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:00.217390 4441 patch_prober.go:29] interesting pod/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 02 22:04:00.217884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:00.217452 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 02 22:04:01.105460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.105425 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:01.217682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.217636 4441 patch_prober.go:29] interesting pod/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 02 22:04:01.218238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.217721 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 02 22:04:01.496793 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.496708 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" podUID="e17dd125-18dd-4446-8dbd-887e19cd0461" containerName="payload-processing" containerID="cri-o://6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4" gracePeriod=25 Nov 02 22:04:01.603231 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.603190 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" podUID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" containerName="payload-pre-processing" containerID="cri-o://91f625f88d3c10c6d1127440a7e527e53a78023a5ee715ae578c25fc8d7b3a81" gracePeriod=25 Nov 02 22:04:01.805190 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.805164 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 22:04:01.931101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.931068 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17dd125-18dd-4446-8dbd-887e19cd0461-kube-api-access-j4wz7\" (UniqueName: \"kubernetes.io/projected/e17dd125-18dd-4446-8dbd-887e19cd0461-kube-api-access-j4wz7\") pod \"e17dd125-18dd-4446-8dbd-887e19cd0461\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " Nov 02 22:04:01.931300 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.931158 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17dd125-18dd-4446-8dbd-887e19cd0461-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17dd125-18dd-4446-8dbd-887e19cd0461-plugins-config-volume\") pod \"e17dd125-18dd-4446-8dbd-887e19cd0461\" (UID: \"e17dd125-18dd-4446-8dbd-887e19cd0461\") " Nov 02 22:04:01.931609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.931581 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17dd125-18dd-4446-8dbd-887e19cd0461-plugins-config-volume" pod "e17dd125-18dd-4446-8dbd-887e19cd0461" (UID: "e17dd125-18dd-4446-8dbd-887e19cd0461"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:01.931841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.931809 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17dd125-18dd-4446-8dbd-887e19cd0461-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:01.933232 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:01.933209 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17dd125-18dd-4446-8dbd-887e19cd0461-kube-api-access-j4wz7" pod "e17dd125-18dd-4446-8dbd-887e19cd0461" (UID: "e17dd125-18dd-4446-8dbd-887e19cd0461"). InnerVolumeSpecName "kube-api-access-j4wz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:02.033798 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.033702 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wz7\" (UniqueName: \"kubernetes.io/projected/e17dd125-18dd-4446-8dbd-887e19cd0461-kube-api-access-j4wz7\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:02.216881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.216843 4441 patch_prober.go:29] interesting pod/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 02 22:04:02.217085 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.216899 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 02 22:04:02.630013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.629961 4441 generic.go:356] "Generic (PLEG): container finished" podID="e17dd125-18dd-4446-8dbd-887e19cd0461" containerID="6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4" exitCode=0 Nov 02 22:04:02.630529 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.630036 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" Nov 02 22:04:02.630529 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.630070 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" event={"ID":"e17dd125-18dd-4446-8dbd-887e19cd0461","Type":"ContainerDied","Data":"6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4"} Nov 02 22:04:02.630529 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.630139 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72" event={"ID":"e17dd125-18dd-4446-8dbd-887e19cd0461","Type":"ContainerDied","Data":"348014309d21badbb8e8195cee2be0bd86a0a4f264bd5d92c7bdd7ea549db238"} Nov 02 22:04:02.630529 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.630174 4441 scope.go:122] "RemoveContainer" containerID="6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4" Nov 02 22:04:02.640594 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.640574 4441 scope.go:122] "RemoveContainer" containerID="6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4" Nov 02 22:04:02.640898 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:02.640876 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4\": container with ID starting with 6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4 not found: ID does not exist" containerID="6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4" Nov 02 22:04:02.640984 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.640909 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4"} err="failed to get container status \"6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4\": rpc error: code = NotFound desc = could not find container \"6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4\": container with ID starting with 6d7779bf4e67008f6eb33de5829603f8d5512db0eb57978eea6d2466fdbc58d4 not found: ID does not exist" Nov 02 22:04:02.666281 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.666237 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:02.797866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.797824 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72"] Nov 02 22:04:02.804789 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.804761 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:04:02.841748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.841712 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70d237-c848cd89-hph72"] Nov 02 22:04:02.896938 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:02.896552 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:03.017779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:03.017739 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:03.146266 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:03.146235 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x"] Nov 02 22:04:03.149652 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:03.149592 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcb857-t264x"] Nov 02 22:04:03.217011 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:03.216951 4441 patch_prober.go:29] interesting pod/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 02 22:04:03.217187 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:03.217044 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 02 22:04:03.562769 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:03.562739 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:04.221449 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:04.221409 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:04:04.222810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:04.222787 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:04:04.584141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:04.584108 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459bc042-b104-4d52-9f0b-e6d2d0bcc7e2" path="/var/lib/kubelet/pods/459bc042-b104-4d52-9f0b-e6d2d0bcc7e2/volumes" Nov 02 22:04:04.584539 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:04.584525 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17dd125-18dd-4446-8dbd-887e19cd0461" path="/var/lib/kubelet/pods/e17dd125-18dd-4446-8dbd-887e19cd0461/volumes" Nov 02 22:04:06.403175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:06.403137 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" podUID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:07.688779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:07.688729 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:09.250357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:09.250310 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:09.377094 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:09.377063 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:09.568230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:09.568197 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:10.030512 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.030424 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:10.585861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.585822 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv"] Nov 02 22:04:10.586542 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586521 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e17dd125-18dd-4446-8dbd-887e19cd0461" containerName="payload-processing" Nov 02 22:04:10.586609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586545 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17dd125-18dd-4446-8dbd-887e19cd0461" containerName="payload-processing" Nov 02 22:04:10.586609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586601 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b3f7af50-2c16-4253-9666-9b20fcc311b7" containerName="payload-pre-processing" Nov 02 22:04:10.586682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586609 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f7af50-2c16-4253-9666-9b20fcc311b7" containerName="payload-pre-processing" Nov 02 22:04:10.586682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586625 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cabaa10f-0506-45c5-ab4f-4e9a10aa3216" containerName="revoke-keys" Nov 02 22:04:10.586682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586633 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabaa10f-0506-45c5-ab4f-4e9a10aa3216" containerName="revoke-keys" Nov 02 22:04:10.586783 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586750 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f7af50-2c16-4253-9666-9b20fcc311b7" containerName="payload-pre-processing" Nov 02 22:04:10.586783 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586764 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabaa10f-0506-45c5-ab4f-4e9a10aa3216" containerName="revoke-keys" Nov 02 22:04:10.586783 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.586779 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17dd125-18dd-4446-8dbd-887e19cd0461" containerName="payload-processing" Nov 02 22:04:10.591317 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.591293 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.616440 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:10.616401 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:04:10.640833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.640807 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv"] Nov 02 22:04:10.683415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.683387 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:04:10.683415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.683387 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:04:10.683595 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.683424 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:04:10.687565 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.687551 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:04:10.724244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.724215 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.724381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.724276 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.724381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.724359 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpht\" (UniqueName: \"kubernetes.io/projected/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-kube-api-access-vmpht\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.825570 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.825543 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.825704 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.825588 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpht\" (UniqueName: \"kubernetes.io/projected/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-kube-api-access-vmpht\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.825704 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.825653 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.826335 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.826306 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.828029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.828005 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.879673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.879611 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpht\" (UniqueName: \"kubernetes.io/projected/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-kube-api-access-vmpht\") pod \"maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:10.904914 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:10.904889 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:11.275596 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:11.275557 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv"] Nov 02 22:04:11.275949 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:11.275925 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc654ce1d_3199_41c4_a4a1_32e6cde79fc6.slice/crio-24c9b2ef275da99184c5bf58dfebbd44c3643fc6c522df4be6172720b874ae0e WatchSource:0}: Error finding container 24c9b2ef275da99184c5bf58dfebbd44c3643fc6c522df4be6172720b874ae0e: Status 404 returned error can't find the container with id 24c9b2ef275da99184c5bf58dfebbd44c3643fc6c522df4be6172720b874ae0e Nov 02 22:04:11.710153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:11.710119 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" event={"ID":"c654ce1d-3199-41c4-a4a1-32e6cde79fc6","Type":"ContainerStarted","Data":"c9db58053a755458779097c5ab6f28381d89c2aed3386533cb3499244f36f9b6"} Nov 02 22:04:11.710516 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:11.710170 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" event={"ID":"c654ce1d-3199-41c4-a4a1-32e6cde79fc6","Type":"ContainerStarted","Data":"24c9b2ef275da99184c5bf58dfebbd44c3643fc6c522df4be6172720b874ae0e"} Nov 02 22:04:11.818888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:11.818834 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" podStartSLOduration=1.818820136 podStartE2EDuration="1.818820136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:11.817138013 +0000 UTC m=+3081.372766351" watchObservedRunningTime="2026-11-02 22:04:11.818820136 +0000 UTC m=+3081.374448473" Nov 02 22:04:11.857853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:11.857764 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:12.720049 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:12.720010 4441 generic.go:356] "Generic (PLEG): container finished" podID="c654ce1d-3199-41c4-a4a1-32e6cde79fc6" containerID="c9db58053a755458779097c5ab6f28381d89c2aed3386533cb3499244f36f9b6" exitCode=0 Nov 02 22:04:12.720560 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:12.720086 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" event={"ID":"c654ce1d-3199-41c4-a4a1-32e6cde79fc6","Type":"ContainerDied","Data":"c9db58053a755458779097c5ab6f28381d89c2aed3386533cb3499244f36f9b6"} Nov 02 22:04:13.316120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:13.316088 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:13.625327 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:13.625229 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:13.741455 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:13.741422 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:13.837402 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:13.837367 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh"] Nov 02 22:04:13.878996 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:13.878910 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:13.882218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:13.882199 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f22129-kxswh"] Nov 02 22:04:14.063016 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.062979 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-openshift-ca\") pod \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " Nov 02 22:04:14.063215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.063059 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-xks-ca\") pod \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " Nov 02 22:04:14.063215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.063194 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-kube-api-access-vmpht\" (UniqueName: \"kubernetes.io/projected/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-kube-api-access-vmpht\") pod \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\" (UID: \"c654ce1d-3199-41c4-a4a1-32e6cde79fc6\") " Nov 02 22:04:14.063408 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.063382 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-openshift-ca" pod "c654ce1d-3199-41c4-a4a1-32e6cde79fc6" (UID: "c654ce1d-3199-41c4-a4a1-32e6cde79fc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:14.063663 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.063641 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:14.065333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.065307 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-xks-ca" pod "c654ce1d-3199-41c4-a4a1-32e6cde79fc6" (UID: "c654ce1d-3199-41c4-a4a1-32e6cde79fc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:14.065463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.065433 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-kube-api-access-vmpht" pod "c654ce1d-3199-41c4-a4a1-32e6cde79fc6" (UID: "c654ce1d-3199-41c4-a4a1-32e6cde79fc6"). InnerVolumeSpecName "kube-api-access-vmpht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:14.165069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.164985 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmpht\" (UniqueName: \"kubernetes.io/projected/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-kube-api-access-vmpht\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:14.165069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.165021 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c654ce1d-3199-41c4-a4a1-32e6cde79fc6-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:14.227550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.227515 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:14.585220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.585187 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e23bd79-ecbc-46ba-9173-246993fddf1b" path="/var/lib/kubelet/pods/2e23bd79-ecbc-46ba-9173-246993fddf1b/volumes" Nov 02 22:04:14.739437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.739407 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" Nov 02 22:04:14.739630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.739408 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv" event={"ID":"c654ce1d-3199-41c4-a4a1-32e6cde79fc6","Type":"ContainerDied","Data":"24c9b2ef275da99184c5bf58dfebbd44c3643fc6c522df4be6172720b874ae0e"} Nov 02 22:04:14.739630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.739532 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c9b2ef275da99184c5bf58dfebbd44c3643fc6c522df4be6172720b874ae0e" Nov 02 22:04:14.990275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:14.990191 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z"] Nov 02 22:04:15.183844 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:15.183793 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:16.404121 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:16.404079 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" podUID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:17.800487 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:17.800448 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:19.408139 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:19.408104 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:19.603237 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:19.603196 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:19.739416 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:19.739326 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:19.889058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:19.889025 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr"] Nov 02 22:04:19.935682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:19.935653 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2dd3b-gsszr"] Nov 02 22:04:20.247402 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:20.247368 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:20.383076 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:20.383047 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:20.518018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:20.517924 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:20.584091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:20.584062 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b302d04-e3f2-4c17-8f08-e1939d74f8e0" path="/var/lib/kubelet/pods/3b302d04-e3f2-4c17-8f08-e1939d74f8e0/volumes" Nov 02 22:04:20.723337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:20.723294 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:21.158371 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.158332 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:21.626321 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.626296 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt"] Nov 02 22:04:21.626847 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.626826 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c654ce1d-3199-41c4-a4a1-32e6cde79fc6" containerName="revoke-keys" Nov 02 22:04:21.626847 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.626848 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c654ce1d-3199-41c4-a4a1-32e6cde79fc6" containerName="revoke-keys" Nov 02 22:04:21.627047 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.627002 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c654ce1d-3199-41c4-a4a1-32e6cde79fc6" containerName="revoke-keys" Nov 02 22:04:21.630440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.630422 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.632320 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.632300 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:21.660303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.660274 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.660416 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.660339 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ml8\" (UniqueName: \"kubernetes.io/projected/a3d2822f-3714-4402-a11b-2798ebb41753-kube-api-access-f9ml8\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.660416 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.660405 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.683601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.683575 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt"] Nov 02 22:04:21.725753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.725725 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:04:21.725753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.725746 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:04:21.726010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.725824 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:04:21.727102 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.727085 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:04:21.762078 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.762050 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.762238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.762108 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ml8\" (UniqueName: \"kubernetes.io/projected/a3d2822f-3714-4402-a11b-2798ebb41753-kube-api-access-f9ml8\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.762238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.762161 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.762764 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.762739 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.764470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.764449 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.817026 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.816948 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ml8\" (UniqueName: \"kubernetes.io/projected/a3d2822f-3714-4402-a11b-2798ebb41753-kube-api-access-f9ml8\") pod \"maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:21.837673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.837635 4441 generic.go:356] "Generic (PLEG): container finished" podID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" containerID="91f625f88d3c10c6d1127440a7e527e53a78023a5ee715ae578c25fc8d7b3a81" exitCode=0 Nov 02 22:04:21.837822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.837703 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" event={"ID":"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8","Type":"ContainerDied","Data":"91f625f88d3c10c6d1127440a7e527e53a78023a5ee715ae578c25fc8d7b3a81"} Nov 02 22:04:21.889460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.889433 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 22:04:21.964165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.964140 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-plugins-config-volume\") pod \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " Nov 02 22:04:21.964337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.964181 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-kube-api-access-rkd6w\" (UniqueName: \"kubernetes.io/projected/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-kube-api-access-rkd6w\") pod \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\" (UID: \"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8\") " Nov 02 22:04:21.964500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.964479 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-plugins-config-volume" pod "0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" (UID: "0eaaff1d-b8c5-44b4-9fa5-f44e795575f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:21.964623 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.964605 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.966271 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:21.966253 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-kube-api-access-rkd6w" pod "0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" (UID: "0eaaff1d-b8c5-44b4-9fa5-f44e795575f8"). InnerVolumeSpecName "kube-api-access-rkd6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:22.001766 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.001727 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:22.065671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.065644 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkd6w\" (UniqueName: \"kubernetes.io/projected/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8-kube-api-access-rkd6w\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:22.374721 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.374689 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt"] Nov 02 22:04:22.376036 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:22.376010 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d2822f_3714_4402_a11b_2798ebb41753.slice/crio-49b9198cfc5f3dd0ced8e80a77a70192a1aabf521a8dcecb86dcc58532b6c9b8 WatchSource:0}: Error finding container 49b9198cfc5f3dd0ced8e80a77a70192a1aabf521a8dcecb86dcc58532b6c9b8: Status 404 returned error can't find the container with id 49b9198cfc5f3dd0ced8e80a77a70192a1aabf521a8dcecb86dcc58532b6c9b8 Nov 02 22:04:22.848604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.848572 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" Nov 02 22:04:22.848604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.848590 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg" event={"ID":"0eaaff1d-b8c5-44b4-9fa5-f44e795575f8","Type":"ContainerDied","Data":"a50811470d06c9d77519821a9cc235af13354788ee92fbfcb0f8272f1c66d9f0"} Nov 02 22:04:22.849133 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.848660 4441 scope.go:122] "RemoveContainer" containerID="91f625f88d3c10c6d1127440a7e527e53a78023a5ee715ae578c25fc8d7b3a81" Nov 02 22:04:22.850455 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.850424 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" event={"ID":"a3d2822f-3714-4402-a11b-2798ebb41753","Type":"ContainerStarted","Data":"99237213e465b23d4e8e72a60ba5d4dd602de47c28119a38fe5e352cb7fed89f"} Nov 02 22:04:22.850571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:22.850484 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" event={"ID":"a3d2822f-3714-4402-a11b-2798ebb41753","Type":"ContainerStarted","Data":"49b9198cfc5f3dd0ced8e80a77a70192a1aabf521a8dcecb86dcc58532b6c9b8"} Nov 02 22:04:23.006542 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.006443 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" podStartSLOduration=2.006428775 podStartE2EDuration="2.006428775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:23.00450054 +0000 UTC m=+3092.560128911" watchObservedRunningTime="2026-11-02 22:04:23.006428775 +0000 UTC m=+3092.562057139" Nov 02 22:04:23.111427 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.111394 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg"] Nov 02 22:04:23.117572 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.117545 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70d237-7ddc758bff-6kzcg"] Nov 02 22:04:23.629355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.629318 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq"] Nov 02 22:04:23.806194 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.806159 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:23.864686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.864653 4441 generic.go:356] "Generic (PLEG): container finished" podID="a3d2822f-3714-4402-a11b-2798ebb41753" containerID="99237213e465b23d4e8e72a60ba5d4dd602de47c28119a38fe5e352cb7fed89f" exitCode=0 Nov 02 22:04:23.865134 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.864798 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" event={"ID":"a3d2822f-3714-4402-a11b-2798ebb41753","Type":"ContainerDied","Data":"99237213e465b23d4e8e72a60ba5d4dd602de47c28119a38fe5e352cb7fed89f"} Nov 02 22:04:23.945051 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:23.944698 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:24.086563 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:24.086532 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:24.587397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:24.587361 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" path="/var/lib/kubelet/pods/0eaaff1d-b8c5-44b4-9fa5-f44e795575f8/volumes" Nov 02 22:04:25.035133 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.035111 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:25.105959 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.105921 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-openshift-ca\") pod \"a3d2822f-3714-4402-a11b-2798ebb41753\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " Nov 02 22:04:25.106167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.105964 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-xks-ca\") pod \"a3d2822f-3714-4402-a11b-2798ebb41753\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " Nov 02 22:04:25.106167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.106006 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d2822f-3714-4402-a11b-2798ebb41753-kube-api-access-f9ml8\" (UniqueName: \"kubernetes.io/projected/a3d2822f-3714-4402-a11b-2798ebb41753-kube-api-access-f9ml8\") pod \"a3d2822f-3714-4402-a11b-2798ebb41753\" (UID: \"a3d2822f-3714-4402-a11b-2798ebb41753\") " Nov 02 22:04:25.106585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.106548 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-openshift-ca" pod "a3d2822f-3714-4402-a11b-2798ebb41753" (UID: "a3d2822f-3714-4402-a11b-2798ebb41753"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:25.108770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.108738 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-xks-ca" pod "a3d2822f-3714-4402-a11b-2798ebb41753" (UID: "a3d2822f-3714-4402-a11b-2798ebb41753"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:25.110477 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.110452 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d2822f-3714-4402-a11b-2798ebb41753-kube-api-access-f9ml8" pod "a3d2822f-3714-4402-a11b-2798ebb41753" (UID: "a3d2822f-3714-4402-a11b-2798ebb41753"). InnerVolumeSpecName "kube-api-access-f9ml8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:25.207062 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.206991 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:25.207062 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.207016 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d2822f-3714-4402-a11b-2798ebb41753-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:25.207062 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.207025 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ml8\" (UniqueName: \"kubernetes.io/projected/a3d2822f-3714-4402-a11b-2798ebb41753-kube-api-access-f9ml8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:25.435117 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.435079 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:25.706409 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.706377 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:25.824293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.824256 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:25.881587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.881560 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" Nov 02 22:04:25.881767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.881592 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt" event={"ID":"a3d2822f-3714-4402-a11b-2798ebb41753","Type":"ContainerDied","Data":"49b9198cfc5f3dd0ced8e80a77a70192a1aabf521a8dcecb86dcc58532b6c9b8"} Nov 02 22:04:25.881767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.881662 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49b9198cfc5f3dd0ced8e80a77a70192a1aabf521a8dcecb86dcc58532b6c9b8" Nov 02 22:04:25.908105 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.908075 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld"] Nov 02 22:04:25.911405 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:25.911385 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70d237-n5qld"] Nov 02 22:04:26.071222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.071139 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:26.174909 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.174874 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc"] Nov 02 22:04:26.192097 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.192061 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9"] Nov 02 22:04:26.223281 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.223249 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:26.356161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.356074 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp"] Nov 02 22:04:26.356332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.356307 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" containerID="cri-o://63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e" gracePeriod=30 Nov 02 22:04:26.359122 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.359091 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:26.484550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.484509 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:26.593775 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:26.593734 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabaa10f-0506-45c5-ab4f-4e9a10aa3216" path="/var/lib/kubelet/pods/cabaa10f-0506-45c5-ab4f-4e9a10aa3216/volumes" Nov 02 22:04:27.669222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:27.669191 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:28.629687 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:28.629644 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" podUID="c7fd939e-a6b3-4579-b84d-c6b7da153315" containerName="payload-pre-processing" containerID="cri-o://b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148" gracePeriod=25 Nov 02 22:04:28.916112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:28.916018 4441 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:04:28.916112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:28.916071 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:04:30.010124 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.010064 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" containerID="cri-o://d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5" gracePeriod=45 Nov 02 22:04:30.278528 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.278463 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 22:04:30.379009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.378964 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-tmp-dir\") pod \"f69caa72-3429-40e8-886e-0a97015ef6e9\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " Nov 02 22:04:30.379203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379036 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-dshm\") pod \"f69caa72-3429-40e8-886e-0a97015ef6e9\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " Nov 02 22:04:30.379203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379075 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-model-cache\") pod \"f69caa72-3429-40e8-886e-0a97015ef6e9\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " Nov 02 22:04:30.379203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379124 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69caa72-3429-40e8-886e-0a97015ef6e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f69caa72-3429-40e8-886e-0a97015ef6e9-tls-certs\") pod \"f69caa72-3429-40e8-886e-0a97015ef6e9\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " Nov 02 22:04:30.379203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379154 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69caa72-3429-40e8-886e-0a97015ef6e9-kube-api-access-d5r6v\" (UniqueName: \"kubernetes.io/projected/f69caa72-3429-40e8-886e-0a97015ef6e9-kube-api-access-d5r6v\") pod \"f69caa72-3429-40e8-886e-0a97015ef6e9\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " Nov 02 22:04:30.379203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379190 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-home\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-home\") pod \"f69caa72-3429-40e8-886e-0a97015ef6e9\" (UID: \"f69caa72-3429-40e8-886e-0a97015ef6e9\") " Nov 02 22:04:30.379534 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379243 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-tmp-dir" pod "f69caa72-3429-40e8-886e-0a97015ef6e9" (UID: "f69caa72-3429-40e8-886e-0a97015ef6e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.379534 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379353 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-model-cache" pod "f69caa72-3429-40e8-886e-0a97015ef6e9" (UID: "f69caa72-3429-40e8-886e-0a97015ef6e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.379731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379681 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.379731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379707 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.379963 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.379946 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-home" pod "f69caa72-3429-40e8-886e-0a97015ef6e9" (UID: "f69caa72-3429-40e8-886e-0a97015ef6e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.382116 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.382077 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-dshm" pod "f69caa72-3429-40e8-886e-0a97015ef6e9" (UID: "f69caa72-3429-40e8-886e-0a97015ef6e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.382237 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.382174 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69caa72-3429-40e8-886e-0a97015ef6e9-tls-certs" pod "f69caa72-3429-40e8-886e-0a97015ef6e9" (UID: "f69caa72-3429-40e8-886e-0a97015ef6e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:30.382237 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.382207 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69caa72-3429-40e8-886e-0a97015ef6e9-kube-api-access-d5r6v" pod "f69caa72-3429-40e8-886e-0a97015ef6e9" (UID: "f69caa72-3429-40e8-886e-0a97015ef6e9"). InnerVolumeSpecName "kube-api-access-d5r6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:30.481448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.481404 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.481618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.481452 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69caa72-3429-40e8-886e-0a97015ef6e9-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.481618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.481474 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5r6v\" (UniqueName: \"kubernetes.io/projected/f69caa72-3429-40e8-886e-0a97015ef6e9-kube-api-access-d5r6v\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.481618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.481495 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69caa72-3429-40e8-886e-0a97015ef6e9-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.708239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.708201 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:30.918580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.918488 4441 generic.go:356] "Generic (PLEG): container finished" podID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerID="d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5" exitCode=0 Nov 02 22:04:30.918580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.918542 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" Nov 02 22:04:30.918790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.918576 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" event={"ID":"f69caa72-3429-40e8-886e-0a97015ef6e9","Type":"ContainerDied","Data":"d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5"} Nov 02 22:04:30.918790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.918627 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z" event={"ID":"f69caa72-3429-40e8-886e-0a97015ef6e9","Type":"ContainerDied","Data":"68927e1c9637276510b9b26194af561b3e38b590b5e1b11707be403ea60f2e96"} Nov 02 22:04:30.918790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.918648 4441 scope.go:122] "RemoveContainer" containerID="d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5" Nov 02 22:04:30.928876 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.928857 4441 scope.go:122] "RemoveContainer" containerID="d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5" Nov 02 22:04:30.929174 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:30.929151 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5\": container with ID starting with d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5 not found: ID does not exist" containerID="d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5" Nov 02 22:04:30.929275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:30.929188 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5"} err="failed to get container status \"d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5\": rpc error: code = NotFound desc = could not find container \"d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5\": container with ID starting with d7da825dbc012084cc8bae08c397ccc19bc8c09ab79ef197195de9f2e9dbabe5 not found: ID does not exist" Nov 02 22:04:31.074924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.074890 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z"] Nov 02 22:04:31.121772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.121731 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-w3-9a6cf2-kserve-84856cb4bc-r7q8z"] Nov 02 22:04:31.636922 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.636857 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 22:04:31.795267 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795232 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-certs\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795450 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795326 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-kube-api-access-fxmz6\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-kube-api-access-fxmz6\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795450 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795376 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-token\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-token\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795450 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795433 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istiod-ca-cert\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795477 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-data\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795537 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-certs" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:31.795636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795556 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-socket\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795583 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-envoy\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795655 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-credential-socket\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795694 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-podinfo\") pod \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\" (UID: \"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95\") " Nov 02 22:04:31.795864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795781 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-data" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:31.796071 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795872 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-socket" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:31.796071 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.795900 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istiod-ca-cert" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:31.796196 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.796184 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.796256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.796205 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.796256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.796217 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.796256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.796229 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.796256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.796231 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-credential-socket" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:31.797776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.797750 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-token" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:31.797776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.797761 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-kube-api-access-fxmz6" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "kube-api-access-fxmz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:31.798192 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.798175 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-podinfo" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:04:31.798243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.798192 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-envoy" pod "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" (UID: "972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:31.897664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.897599 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxmz6\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-kube-api-access-fxmz6\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.897664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.897625 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.897664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.897635 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.897664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.897644 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.897664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.897652 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:31.928172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.928105 4441 generic.go:356] "Generic (PLEG): container finished" podID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerID="63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e" exitCode=0 Nov 02 22:04:31.928172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.928154 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" Nov 02 22:04:31.928357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.928180 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" event={"ID":"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95","Type":"ContainerDied","Data":"63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e"} Nov 02 22:04:31.928357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.928228 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp" event={"ID":"972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95","Type":"ContainerDied","Data":"af254792a4952c1cbfb08c3b5c35dc6a43407ad247efe039aa87dac7e2e0f98d"} Nov 02 22:04:31.928357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.928253 4441 scope.go:122] "RemoveContainer" containerID="63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e" Nov 02 22:04:31.939054 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.939038 4441 scope.go:122] "RemoveContainer" containerID="02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e" Nov 02 22:04:31.947707 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.947690 4441 scope.go:122] "RemoveContainer" containerID="63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e" Nov 02 22:04:31.947984 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:31.947945 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e\": container with ID starting with 63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e not found: ID does not exist" containerID="63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e" Nov 02 22:04:31.948042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.947988 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e"} err="failed to get container status \"63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e\": rpc error: code = NotFound desc = could not find container \"63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e\": container with ID starting with 63869529032d385a9c6da72750e98dcf90bef9707bc20894b7575e322fbefd4e not found: ID does not exist" Nov 02 22:04:31.948042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.948009 4441 scope.go:122] "RemoveContainer" containerID="02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e" Nov 02 22:04:31.948225 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:31.948209 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e\": container with ID starting with 02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e not found: ID does not exist" containerID="02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e" Nov 02 22:04:31.948274 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:31.948229 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e"} err="failed to get container status \"02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e\": rpc error: code = NotFound desc = could not find container \"02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e\": container with ID starting with 02cbed8b3302dddca98cb94f64eb32f45e3bfdb3c146cdc086a2f00cd2b4fd4e not found: ID does not exist" Nov 02 22:04:32.085933 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.085903 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp"] Nov 02 22:04:32.089329 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.089304 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-lmksp"] Nov 02 22:04:32.178436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.178360 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:32.313360 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.313329 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:32.485916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.485837 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n"] Nov 02 22:04:32.486145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.486117 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" containerID="cri-o://edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218" gracePeriod=30 Nov 02 22:04:32.518076 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.518040 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:32.588117 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.588078 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" path="/var/lib/kubelet/pods/972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95/volumes" Nov 02 22:04:32.588811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.588759 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" path="/var/lib/kubelet/pods/f69caa72-3429-40e8-886e-0a97015ef6e9/volumes" Nov 02 22:04:32.965762 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:32.965706 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:33.093527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.093496 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq"] Nov 02 22:04:33.198261 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.198224 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg"] Nov 02 22:04:33.420835 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.420802 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" podUID="c7fd939e-a6b3-4579-b84d-c6b7da153315" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:33.428012 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.427955 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv"] Nov 02 22:04:33.428504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428485 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a3d2822f-3714-4402-a11b-2798ebb41753" containerName="revoke-keys" Nov 02 22:04:33.428608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428505 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d2822f-3714-4402-a11b-2798ebb41753" containerName="revoke-keys" Nov 02 22:04:33.428608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428538 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" containerName="payload-pre-processing" Nov 02 22:04:33.428608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428548 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" containerName="payload-pre-processing" Nov 02 22:04:33.428608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428582 4441 cpu_manager.go:414] "containerMap: removing container" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" Nov 02 22:04:33.428608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428589 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428610 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428619 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428643 4441 cpu_manager.go:414] "containerMap: removing container" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428651 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428774 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69caa72-3429-40e8-886e-0a97015ef6e9" containerName="main" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428788 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428798 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d2822f-3714-4402-a11b-2798ebb41753" containerName="revoke-keys" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428811 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eaaff1d-b8c5-44b4-9fa5-f44e795575f8" containerName="payload-pre-processing" Nov 02 22:04:33.428860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.428822 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="972c1f9c-7782-4cf3-9d79-9a7a7a0dfd95" containerName="istio-proxy" Nov 02 22:04:33.433797 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.433776 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.442905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.442318 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:33.482933 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.482902 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:04:33.483465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.483444 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:04:33.483591 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.483484 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:04:33.483591 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.483551 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:04:33.484721 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.484698 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv"] Nov 02 22:04:33.516314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.516282 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.516462 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.516333 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtppx\" (UniqueName: \"kubernetes.io/projected/768bfa6e-d313-466d-a13c-66ab8ef2de80-kube-api-access-jtppx\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.516537 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.516462 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.581383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.581352 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:33.617271 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.617236 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.617449 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.617291 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtppx\" (UniqueName: \"kubernetes.io/projected/768bfa6e-d313-466d-a13c-66ab8ef2de80-kube-api-access-jtppx\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.617449 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.617349 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.618145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.618107 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.620389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.620362 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.672603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.672520 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtppx\" (UniqueName: \"kubernetes.io/projected/768bfa6e-d313-466d-a13c-66ab8ef2de80-kube-api-access-jtppx\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:33.751113 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:33.751084 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:34.120622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.120599 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv"] Nov 02 22:04:34.122313 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:34.122280 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768bfa6e_d313_466d_a13c_66ab8ef2de80.slice/crio-a5d9af2c56f5d271a9cdcc998623216451493f0e3d4a46d590422c8988dd0302 WatchSource:0}: Error finding container a5d9af2c56f5d271a9cdcc998623216451493f0e3d4a46d590422c8988dd0302: Status 404 returned error can't find the container with id a5d9af2c56f5d271a9cdcc998623216451493f0e3d4a46d590422c8988dd0302 Nov 02 22:04:34.872764 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.872735 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz"] Nov 02 22:04:34.876730 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.876712 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:34.926742 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.926717 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp"] Nov 02 22:04:34.930900 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.930878 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:34.954159 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.954133 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" event={"ID":"768bfa6e-d313-466d-a13c-66ab8ef2de80","Type":"ContainerStarted","Data":"47835429f18246970e677a271be2f5c18dda1b32ec7e43c80b94a31845e42782"} Nov 02 22:04:34.954308 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.954173 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" event={"ID":"768bfa6e-d313-466d-a13c-66ab8ef2de80","Type":"ContainerStarted","Data":"a5d9af2c56f5d271a9cdcc998623216451493f0e3d4a46d590422c8988dd0302"} Nov 02 22:04:34.975120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.975093 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz"] Nov 02 22:04:34.984236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:34.984215 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp"] Nov 02 22:04:35.032622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.032590 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.032779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.032635 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhnsk\" (UniqueName: \"kubernetes.io/projected/f90a86e1-903e-405b-a30f-13c0d73e68b3-kube-api-access-lhnsk\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.032779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.032661 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdbr\" (UniqueName: \"kubernetes.io/projected/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-kube-api-access-zhdbr\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.032900 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.032839 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.032949 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.032908 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.032949 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.032942 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.134503 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.134442 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.134503 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.134488 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhnsk\" (UniqueName: \"kubernetes.io/projected/f90a86e1-903e-405b-a30f-13c0d73e68b3-kube-api-access-lhnsk\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.134944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.134519 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdbr\" (UniqueName: \"kubernetes.io/projected/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-kube-api-access-zhdbr\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.134944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.134577 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.134944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.134607 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.134944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.134628 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.135324 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.135297 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.137375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.135391 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.137912 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.137889 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.138218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.138195 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.139063 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.139026 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" podStartSLOduration=2.139014831 podStartE2EDuration="2.139014831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:35.135311357 +0000 UTC m=+3104.690939730" watchObservedRunningTime="2026-11-02 22:04:35.139014831 +0000 UTC m=+3104.694643167" Nov 02 22:04:35.188445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.188415 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdbr\" (UniqueName: \"kubernetes.io/projected/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-kube-api-access-zhdbr\") pod \"maas-api-revoke-sub-2a4babea-l6nbz\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.188445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.188443 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhnsk\" (UniqueName: \"kubernetes.io/projected/f90a86e1-903e-405b-a30f-13c0d73e68b3-kube-api-access-lhnsk\") pod \"maas-api-revoke-sub-ce81ab27-vbhlp\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.190267 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.190251 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:35.244265 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.244052 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:35.555589 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:35.555561 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c3cc8_edf4_4ec4_9bc1_1271d5afa041.slice/crio-2f00e4ce88dba1fde2d81c61976bed0fb54339b5f107f09f17a09302afd16d79 WatchSource:0}: Error finding container 2f00e4ce88dba1fde2d81c61976bed0fb54339b5f107f09f17a09302afd16d79: Status 404 returned error can't find the container with id 2f00e4ce88dba1fde2d81c61976bed0fb54339b5f107f09f17a09302afd16d79 Nov 02 22:04:35.556240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.556216 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz"] Nov 02 22:04:35.657175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.657148 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp"] Nov 02 22:04:35.657703 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:35.657679 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90a86e1_903e_405b_a30f_13c0d73e68b3.slice/crio-3282562a24a0df867784f25b1a713731f327f9c5734991515777ea71fc711fa7 WatchSource:0}: Error finding container 3282562a24a0df867784f25b1a713731f327f9c5734991515777ea71fc711fa7: Status 404 returned error can't find the container with id 3282562a24a0df867784f25b1a713731f327f9c5734991515777ea71fc711fa7 Nov 02 22:04:35.820964 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.820941 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:35.963333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.963304 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" event={"ID":"f90a86e1-903e-405b-a30f-13c0d73e68b3","Type":"ContainerStarted","Data":"3282562a24a0df867784f25b1a713731f327f9c5734991515777ea71fc711fa7"} Nov 02 22:04:35.964871 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.964845 4441 generic.go:356] "Generic (PLEG): container finished" podID="768bfa6e-d313-466d-a13c-66ab8ef2de80" containerID="47835429f18246970e677a271be2f5c18dda1b32ec7e43c80b94a31845e42782" exitCode=0 Nov 02 22:04:35.964991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.964925 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" event={"ID":"768bfa6e-d313-466d-a13c-66ab8ef2de80","Type":"ContainerDied","Data":"47835429f18246970e677a271be2f5c18dda1b32ec7e43c80b94a31845e42782"} Nov 02 22:04:35.966444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.966423 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" event={"ID":"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041","Type":"ContainerStarted","Data":"b7f2cb81764c8ccd46d4c43736486434be71768d5801a56561d9792454be48b5"} Nov 02 22:04:35.966558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:35.966456 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" event={"ID":"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041","Type":"ContainerStarted","Data":"2f00e4ce88dba1fde2d81c61976bed0fb54339b5f107f09f17a09302afd16d79"} Nov 02 22:04:36.179452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:36.179356 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" podStartSLOduration=2.179341891 podStartE2EDuration="2.179341891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:36.178056788 +0000 UTC m=+3105.733685126" watchObservedRunningTime="2026-11-02 22:04:36.179341891 +0000 UTC m=+3105.734970228" Nov 02 22:04:36.976939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:36.976904 4441 generic.go:356] "Generic (PLEG): container finished" podID="0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" containerID="b7f2cb81764c8ccd46d4c43736486434be71768d5801a56561d9792454be48b5" exitCode=0 Nov 02 22:04:36.977167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:36.976998 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" event={"ID":"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041","Type":"ContainerDied","Data":"b7f2cb81764c8ccd46d4c43736486434be71768d5801a56561d9792454be48b5"} Nov 02 22:04:36.978842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:36.978816 4441 generic.go:356] "Generic (PLEG): container finished" podID="f90a86e1-903e-405b-a30f-13c0d73e68b3" containerID="f67c613940cf242e8c50cdaa9dcecb3887c4c62820bd7c2163c266803b5bb899" exitCode=0 Nov 02 22:04:36.978997 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:36.978889 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" event={"ID":"f90a86e1-903e-405b-a30f-13c0d73e68b3","Type":"ContainerDied","Data":"f67c613940cf242e8c50cdaa9dcecb3887c4c62820bd7c2163c266803b5bb899"} Nov 02 22:04:37.134107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.134083 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:37.255717 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.255631 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-openshift-ca\") pod \"768bfa6e-d313-466d-a13c-66ab8ef2de80\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " Nov 02 22:04:37.256200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.255768 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-xks-ca\") pod \"768bfa6e-d313-466d-a13c-66ab8ef2de80\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " Nov 02 22:04:37.256200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.255861 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768bfa6e-d313-466d-a13c-66ab8ef2de80-kube-api-access-jtppx\" (UniqueName: \"kubernetes.io/projected/768bfa6e-d313-466d-a13c-66ab8ef2de80-kube-api-access-jtppx\") pod \"768bfa6e-d313-466d-a13c-66ab8ef2de80\" (UID: \"768bfa6e-d313-466d-a13c-66ab8ef2de80\") " Nov 02 22:04:37.256200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.256004 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-openshift-ca" pod "768bfa6e-d313-466d-a13c-66ab8ef2de80" (UID: "768bfa6e-d313-466d-a13c-66ab8ef2de80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:37.256313 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.256250 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.257901 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.257871 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-xks-ca" pod "768bfa6e-d313-466d-a13c-66ab8ef2de80" (UID: "768bfa6e-d313-466d-a13c-66ab8ef2de80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:37.258050 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.257901 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768bfa6e-d313-466d-a13c-66ab8ef2de80-kube-api-access-jtppx" pod "768bfa6e-d313-466d-a13c-66ab8ef2de80" (UID: "768bfa6e-d313-466d-a13c-66ab8ef2de80"). InnerVolumeSpecName "kube-api-access-jtppx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:37.357152 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.357119 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768bfa6e-d313-466d-a13c-66ab8ef2de80-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.357152 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.357149 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtppx\" (UniqueName: \"kubernetes.io/projected/768bfa6e-d313-466d-a13c-66ab8ef2de80-kube-api-access-jtppx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.741940 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.741915 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 22:04:37.861809 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.861775 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c41f4c-73ee-440d-befc-9def36a8b0ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c41f4c-73ee-440d-befc-9def36a8b0ad-istiod-ca-cert\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.861995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.861876 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-token\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-token\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.861995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.861937 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-envoy\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.861995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.861992 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-data\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.862215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862040 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-certs\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.862215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862097 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-socket\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.862215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862139 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-podinfo\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.862434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862223 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-kube-api-access-q5q4b\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-kube-api-access-q5q4b\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.862434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862224 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c41f4c-73ee-440d-befc-9def36a8b0ad-istiod-ca-cert" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:37.862434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862252 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-credential-socket\") pod \"40c41f4c-73ee-440d-befc-9def36a8b0ad\" (UID: \"40c41f4c-73ee-440d-befc-9def36a8b0ad\") " Nov 02 22:04:37.862604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862512 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-certs" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:37.862671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862657 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.862737 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862671 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c41f4c-73ee-440d-befc-9def36a8b0ad-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.862737 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862698 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-data" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:37.862903 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.862881 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-socket" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:37.863106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.863080 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-credential-socket" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:37.864457 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.864430 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-token" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:37.864584 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.864535 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-envoy" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:37.864732 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.864712 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-kube-api-access-q5q4b" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "kube-api-access-q5q4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:37.865152 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.865137 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-podinfo" pod "40c41f4c-73ee-440d-befc-9def36a8b0ad" (UID: "40c41f4c-73ee-440d-befc-9def36a8b0ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:04:37.952264 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.952236 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q"] Nov 02 22:04:37.963345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.963323 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.963345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.963346 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.963535 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.963355 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.963535 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.963366 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.963535 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.963374 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c41f4c-73ee-440d-befc-9def36a8b0ad-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.963535 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.963384 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5q4b\" (UniqueName: \"kubernetes.io/projected/40c41f4c-73ee-440d-befc-9def36a8b0ad-kube-api-access-q5q4b\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.963535 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.963393 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c41f4c-73ee-440d-befc-9def36a8b0ad-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:37.991623 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.991591 4441 generic.go:356] "Generic (PLEG): container finished" podID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerID="edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218" exitCode=0 Nov 02 22:04:37.991763 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.991646 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" Nov 02 22:04:37.991763 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.991644 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" event={"ID":"40c41f4c-73ee-440d-befc-9def36a8b0ad","Type":"ContainerDied","Data":"edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218"} Nov 02 22:04:37.991763 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.991726 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n" event={"ID":"40c41f4c-73ee-440d-befc-9def36a8b0ad","Type":"ContainerDied","Data":"24ead6eb556b44bd3b0308a5fdf69db7ab9f227ab1322ccec92197790f69e5a3"} Nov 02 22:04:37.991953 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.991780 4441 scope.go:122] "RemoveContainer" containerID="edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218" Nov 02 22:04:37.993592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.993573 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" Nov 02 22:04:37.993693 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.993601 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv" event={"ID":"768bfa6e-d313-466d-a13c-66ab8ef2de80","Type":"ContainerDied","Data":"a5d9af2c56f5d271a9cdcc998623216451493f0e3d4a46d590422c8988dd0302"} Nov 02 22:04:37.993693 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:37.993650 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d9af2c56f5d271a9cdcc998623216451493f0e3d4a46d590422c8988dd0302" Nov 02 22:04:38.008225 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.008103 4441 scope.go:122] "RemoveContainer" containerID="5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655" Nov 02 22:04:38.023152 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.023133 4441 scope.go:122] "RemoveContainer" containerID="edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218" Nov 02 22:04:38.023460 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:38.023435 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218\": container with ID starting with edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218 not found: ID does not exist" containerID="edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218" Nov 02 22:04:38.023527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.023474 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218"} err="failed to get container status \"edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218\": rpc error: code = NotFound desc = could not find container \"edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218\": container with ID starting with edcb73093557f67eecbe5a491daf4b03f9802c908ace46c7ad3af044df279218 not found: ID does not exist" Nov 02 22:04:38.023527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.023495 4441 scope.go:122] "RemoveContainer" containerID="5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655" Nov 02 22:04:38.023787 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:38.023767 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655\": container with ID starting with 5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655 not found: ID does not exist" containerID="5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655" Nov 02 22:04:38.023846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.023792 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655"} err="failed to get container status \"5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655\": rpc error: code = NotFound desc = could not find container \"5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655\": container with ID starting with 5e2d0a8f69049705f0fc5567464307419f94ca17265a0aa5bb977985c129e655 not found: ID does not exist" Nov 02 22:04:38.025886 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.025868 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:38.093915 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.093873 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" podUID="f0f4428d-6985-4e19-abef-5e7ce965447a" containerName="payload-processing" containerID="cri-o://d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518" gracePeriod=25 Nov 02 22:04:38.199567 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.199418 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" podUID="a9cffa72-fba9-446c-9780-130d7918e043" containerName="payload-pre-processing" containerID="cri-o://e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c" gracePeriod=25 Nov 02 22:04:38.219910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.219881 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n"] Nov 02 22:04:38.223860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.223834 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-70d237-openshift-default-86895f855-4l98n"] Nov 02 22:04:38.228135 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.228115 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:38.232361 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.232342 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:38.350557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.350534 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 22:04:38.368610 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.368587 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-kube-api-access-zhdbr\" (UniqueName: \"kubernetes.io/projected/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-kube-api-access-zhdbr\") pod \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " Nov 02 22:04:38.368779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.368766 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90a86e1-903e-405b-a30f-13c0d73e68b3-kube-api-access-lhnsk\" (UniqueName: \"kubernetes.io/projected/f90a86e1-903e-405b-a30f-13c0d73e68b3-kube-api-access-lhnsk\") pod \"f90a86e1-903e-405b-a30f-13c0d73e68b3\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " Nov 02 22:04:38.368825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.368811 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-openshift-ca\") pod \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " Nov 02 22:04:38.368867 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.368838 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-openshift-ca\") pod \"f90a86e1-903e-405b-a30f-13c0d73e68b3\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " Nov 02 22:04:38.368867 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.368858 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-xks-ca\") pod \"f90a86e1-903e-405b-a30f-13c0d73e68b3\" (UID: \"f90a86e1-903e-405b-a30f-13c0d73e68b3\") " Nov 02 22:04:38.368987 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.368883 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-xks-ca\") pod \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\" (UID: \"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041\") " Nov 02 22:04:38.369856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.369169 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-openshift-ca" pod "0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" (UID: "0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:38.369856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.369208 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-openshift-ca" pod "f90a86e1-903e-405b-a30f-13c0d73e68b3" (UID: "f90a86e1-903e-405b-a30f-13c0d73e68b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:38.369856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.369348 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.369856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.369367 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.370908 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.370886 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-xks-ca" pod "0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" (UID: "0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:38.371267 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.371241 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-kube-api-access-zhdbr" pod "0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" (UID: "0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041"). InnerVolumeSpecName "kube-api-access-zhdbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:38.371363 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.371276 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-xks-ca" pod "f90a86e1-903e-405b-a30f-13c0d73e68b3" (UID: "f90a86e1-903e-405b-a30f-13c0d73e68b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:38.371501 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.371480 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90a86e1-903e-405b-a30f-13c0d73e68b3-kube-api-access-lhnsk" pod "f90a86e1-903e-405b-a30f-13c0d73e68b3" (UID: "f90a86e1-903e-405b-a30f-13c0d73e68b3"). InnerVolumeSpecName "kube-api-access-lhnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:38.470482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.470406 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f4428d-6985-4e19-abef-5e7ce965447a-kube-api-access-mn95c\" (UniqueName: \"kubernetes.io/projected/f0f4428d-6985-4e19-abef-5e7ce965447a-kube-api-access-mn95c\") pod \"f0f4428d-6985-4e19-abef-5e7ce965447a\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " Nov 02 22:04:38.470482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.470460 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f4428d-6985-4e19-abef-5e7ce965447a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4428d-6985-4e19-abef-5e7ce965447a-plugins-config-volume\") pod \"f0f4428d-6985-4e19-abef-5e7ce965447a\" (UID: \"f0f4428d-6985-4e19-abef-5e7ce965447a\") " Nov 02 22:04:38.470838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.470820 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhdbr\" (UniqueName: \"kubernetes.io/projected/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-kube-api-access-zhdbr\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.470923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.470842 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhnsk\" (UniqueName: \"kubernetes.io/projected/f90a86e1-903e-405b-a30f-13c0d73e68b3-kube-api-access-lhnsk\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.470923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.470856 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90a86e1-903e-405b-a30f-13c0d73e68b3-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.470923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.470870 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.471052 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.470999 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f4428d-6985-4e19-abef-5e7ce965447a-plugins-config-volume" pod "f0f4428d-6985-4e19-abef-5e7ce965447a" (UID: "f0f4428d-6985-4e19-abef-5e7ce965447a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:38.473091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.473069 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f4428d-6985-4e19-abef-5e7ce965447a-kube-api-access-mn95c" pod "f0f4428d-6985-4e19-abef-5e7ce965447a" (UID: "f0f4428d-6985-4e19-abef-5e7ce965447a"). InnerVolumeSpecName "kube-api-access-mn95c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:38.571830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.571791 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn95c\" (UniqueName: \"kubernetes.io/projected/f0f4428d-6985-4e19-abef-5e7ce965447a-kube-api-access-mn95c\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.571830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.571825 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4428d-6985-4e19-abef-5e7ce965447a-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:38.587091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:38.587060 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" path="/var/lib/kubelet/pods/40c41f4c-73ee-440d-befc-9def36a8b0ad/volumes" Nov 02 22:04:39.004880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.004848 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" Nov 02 22:04:39.005081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.004889 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz" event={"ID":"0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041","Type":"ContainerDied","Data":"2f00e4ce88dba1fde2d81c61976bed0fb54339b5f107f09f17a09302afd16d79"} Nov 02 22:04:39.005081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.004951 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f00e4ce88dba1fde2d81c61976bed0fb54339b5f107f09f17a09302afd16d79" Nov 02 22:04:39.006512 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.006486 4441 generic.go:356] "Generic (PLEG): container finished" podID="f0f4428d-6985-4e19-abef-5e7ce965447a" containerID="d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518" exitCode=0 Nov 02 22:04:39.006643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.006534 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" Nov 02 22:04:39.006643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.006563 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" event={"ID":"f0f4428d-6985-4e19-abef-5e7ce965447a","Type":"ContainerDied","Data":"d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518"} Nov 02 22:04:39.006643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.006619 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq" event={"ID":"f0f4428d-6985-4e19-abef-5e7ce965447a","Type":"ContainerDied","Data":"3c368a7fadd637ab37074839f686083825b8856fcc4bf705ab2843498bde48dc"} Nov 02 22:04:39.006822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.006648 4441 scope.go:122] "RemoveContainer" containerID="d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518" Nov 02 22:04:39.010671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.010641 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" Nov 02 22:04:39.010671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.010658 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp" event={"ID":"f90a86e1-903e-405b-a30f-13c0d73e68b3","Type":"ContainerDied","Data":"3282562a24a0df867784f25b1a713731f327f9c5734991515777ea71fc711fa7"} Nov 02 22:04:39.010863 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.010698 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3282562a24a0df867784f25b1a713731f327f9c5734991515777ea71fc711fa7" Nov 02 22:04:39.022073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.022054 4441 scope.go:122] "RemoveContainer" containerID="d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518" Nov 02 22:04:39.022336 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:39.022315 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518\": container with ID starting with d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518 not found: ID does not exist" containerID="d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518" Nov 02 22:04:39.022386 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.022346 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518"} err="failed to get container status \"d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518\": rpc error: code = NotFound desc = could not find container \"d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518\": container with ID starting with d383a7d6f6dafc2f53e4f49355809ad151c2424fdf2580cf4b640bd353059518 not found: ID does not exist" Nov 02 22:04:39.214077 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.214021 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq"] Nov 02 22:04:39.217514 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.217484 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4d9ea0-75b8748dc7-bqkgq"] Nov 02 22:04:39.827864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.827822 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw"] Nov 02 22:04:39.828497 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828476 4441 cpu_manager.go:414] "containerMap: removing container" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" Nov 02 22:04:39.828603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828499 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" Nov 02 22:04:39.828603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828515 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f90a86e1-903e-405b-a30f-13c0d73e68b3" containerName="revoke-keys" Nov 02 22:04:39.828603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828522 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90a86e1-903e-405b-a30f-13c0d73e68b3" containerName="revoke-keys" Nov 02 22:04:39.828603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828560 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" containerName="revoke-keys" Nov 02 22:04:39.828603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828576 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" containerName="revoke-keys" Nov 02 22:04:39.828603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828599 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f0f4428d-6985-4e19-abef-5e7ce965447a" containerName="payload-processing" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828606 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f4428d-6985-4e19-abef-5e7ce965447a" containerName="payload-processing" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828631 4441 cpu_manager.go:414] "containerMap: removing container" podUID="768bfa6e-d313-466d-a13c-66ab8ef2de80" containerName="revoke-keys" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828640 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768bfa6e-d313-466d-a13c-66ab8ef2de80" containerName="revoke-keys" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828761 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828772 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" containerName="revoke-keys" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828786 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f4428d-6985-4e19-abef-5e7ce965447a" containerName="payload-processing" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828799 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="768bfa6e-d313-466d-a13c-66ab8ef2de80" containerName="revoke-keys" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828808 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" Nov 02 22:04:39.828923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.828818 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90a86e1-903e-405b-a30f-13c0d73e68b3" containerName="revoke-keys" Nov 02 22:04:39.833681 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.833660 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:39.884735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.884709 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:04:39.884735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.884715 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:04:39.884992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.884767 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:04:39.884992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.884796 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:04:39.889918 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.889893 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t"] Nov 02 22:04:39.890519 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.890500 4441 cpu_manager.go:414] "containerMap: removing container" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" Nov 02 22:04:39.890562 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.890521 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c41f4c-73ee-440d-befc-9def36a8b0ad" containerName="istio-proxy" Nov 02 22:04:39.894415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.894385 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:39.895321 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.895298 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw"] Nov 02 22:04:39.952403 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.952367 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t"] Nov 02 22:04:39.987677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.987649 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:39.987807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.987689 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:39.987807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.987711 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:39.987807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.987742 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:39.987959 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.987935 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tsd\" (UniqueName: \"kubernetes.io/projected/3782096d-ec58-4a3c-9086-9ee05e10161e-kube-api-access-v2tsd\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:39.988057 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:39.988040 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg9g4\" (UniqueName: \"kubernetes.io/projected/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-kube-api-access-rg9g4\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.088707 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.088616 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsd\" (UniqueName: \"kubernetes.io/projected/3782096d-ec58-4a3c-9086-9ee05e10161e-kube-api-access-v2tsd\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:40.088707 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.088684 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg9g4\" (UniqueName: \"kubernetes.io/projected/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-kube-api-access-rg9g4\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.088940 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.088736 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:40.088940 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.088774 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.088940 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.088802 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:40.088940 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.088838 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.089623 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.089591 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:40.089770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.089591 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.091383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.091355 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.091678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.091647 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:40.142749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.142725 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tsd\" (UniqueName: \"kubernetes.io/projected/3782096d-ec58-4a3c-9086-9ee05e10161e-kube-api-access-v2tsd\") pod \"maas-api-revoke-sub-23eddcca-5ph5t\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:40.142876 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.142755 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg9g4\" (UniqueName: \"kubernetes.io/projected/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-kube-api-access-rg9g4\") pod \"maas-api-revoke-sub-e4cff178-wzvqw\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.146577 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.146562 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:40.216719 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.212944 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:40.518691 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:40.518646 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6ddbb5_059a_4949_9d2e_5d08c29c8d26.slice/crio-23a3729839677a976e743fbf1c4e9ffe2313871c4897e2b23bbce4060493cc08 WatchSource:0}: Error finding container 23a3729839677a976e743fbf1c4e9ffe2313871c4897e2b23bbce4060493cc08: Status 404 returned error can't find the container with id 23a3729839677a976e743fbf1c4e9ffe2313871c4897e2b23bbce4060493cc08 Nov 02 22:04:40.519554 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.519527 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw"] Nov 02 22:04:40.573105 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.573077 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:40.585072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.585045 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f4428d-6985-4e19-abef-5e7ce965447a" path="/var/lib/kubelet/pods/f0f4428d-6985-4e19-abef-5e7ce965447a/volumes" Nov 02 22:04:40.617724 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:40.617677 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:04:40.622131 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:40.622103 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3782096d_ec58_4a3c_9086_9ee05e10161e.slice/crio-5b6bc7b203a82658dc9bc2c65479a7932f384f91120a1dc813560f8fadd0d28e WatchSource:0}: Error finding container 5b6bc7b203a82658dc9bc2c65479a7932f384f91120a1dc813560f8fadd0d28e: Status 404 returned error can't find the container with id 5b6bc7b203a82658dc9bc2c65479a7932f384f91120a1dc813560f8fadd0d28e Nov 02 22:04:40.622635 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:40.622609 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t"] Nov 02 22:04:41.028940 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.028898 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" event={"ID":"3782096d-ec58-4a3c-9086-9ee05e10161e","Type":"ContainerStarted","Data":"3d89d086e4f8c6d197e7d092074b8598c706f2fa24dfaefd8d10accabd0f9a28"} Nov 02 22:04:41.029423 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.028961 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" event={"ID":"3782096d-ec58-4a3c-9086-9ee05e10161e","Type":"ContainerStarted","Data":"5b6bc7b203a82658dc9bc2c65479a7932f384f91120a1dc813560f8fadd0d28e"} Nov 02 22:04:41.030821 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.030753 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" event={"ID":"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26","Type":"ContainerStarted","Data":"b9d0310f8bbe900ca8dfb67317feba294164d0b3637a46903aef0e5e25e5a898"} Nov 02 22:04:41.030821 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.030802 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" event={"ID":"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26","Type":"ContainerStarted","Data":"23a3729839677a976e743fbf1c4e9ffe2313871c4897e2b23bbce4060493cc08"} Nov 02 22:04:41.138645 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.138595 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" podStartSLOduration=2.138581137 podStartE2EDuration="2.138581137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:41.136539659 +0000 UTC m=+3110.692168040" watchObservedRunningTime="2026-11-02 22:04:41.138581137 +0000 UTC m=+3110.694209498" Nov 02 22:04:41.199117 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.199083 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" containerID="cri-o://ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef" gracePeriod=45 Nov 02 22:04:41.228703 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.228674 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" containerID="cri-o://e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06" gracePeriod=45 Nov 02 22:04:41.246441 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.246399 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" podStartSLOduration=2.246385435 podStartE2EDuration="2.246385435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:41.243453488 +0000 UTC m=+3110.799081824" watchObservedRunningTime="2026-11-02 22:04:41.246385435 +0000 UTC m=+3110.802013768" Nov 02 22:04:41.486770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.486741 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 22:04:41.516001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.515975 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 22:04:41.605025 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.604930 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec94be10-a9ef-4259-b450-acd38f34d4e8-kube-api-access-wsgm4\" (UniqueName: \"kubernetes.io/projected/ec94be10-a9ef-4259-b450-acd38f34d4e8-kube-api-access-wsgm4\") pod \"ec94be10-a9ef-4259-b450-acd38f34d4e8\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " Nov 02 22:04:41.605200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.605045 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-dshm\") pod \"ec94be10-a9ef-4259-b450-acd38f34d4e8\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " Nov 02 22:04:41.605200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.605074 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-tmp-dir\") pod \"ec94be10-a9ef-4259-b450-acd38f34d4e8\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " Nov 02 22:04:41.605200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.605132 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-home\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-home\") pod \"ec94be10-a9ef-4259-b450-acd38f34d4e8\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " Nov 02 22:04:41.605394 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.605249 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec94be10-a9ef-4259-b450-acd38f34d4e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94be10-a9ef-4259-b450-acd38f34d4e8-tls-certs\") pod \"ec94be10-a9ef-4259-b450-acd38f34d4e8\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " Nov 02 22:04:41.605394 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.605296 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-model-cache\") pod \"ec94be10-a9ef-4259-b450-acd38f34d4e8\" (UID: \"ec94be10-a9ef-4259-b450-acd38f34d4e8\") " Nov 02 22:04:41.605636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.605576 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-tmp-dir" pod "ec94be10-a9ef-4259-b450-acd38f34d4e8" (UID: "ec94be10-a9ef-4259-b450-acd38f34d4e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.605866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.605727 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.606212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.606124 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-home" pod "ec94be10-a9ef-4259-b450-acd38f34d4e8" (UID: "ec94be10-a9ef-4259-b450-acd38f34d4e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.606212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.606165 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-model-cache" pod "ec94be10-a9ef-4259-b450-acd38f34d4e8" (UID: "ec94be10-a9ef-4259-b450-acd38f34d4e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.608247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.608223 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-dshm" pod "ec94be10-a9ef-4259-b450-acd38f34d4e8" (UID: "ec94be10-a9ef-4259-b450-acd38f34d4e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.608618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.608586 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec94be10-a9ef-4259-b450-acd38f34d4e8-tls-certs" pod "ec94be10-a9ef-4259-b450-acd38f34d4e8" (UID: "ec94be10-a9ef-4259-b450-acd38f34d4e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:41.609584 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.609560 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec94be10-a9ef-4259-b450-acd38f34d4e8-kube-api-access-wsgm4" pod "ec94be10-a9ef-4259-b450-acd38f34d4e8" (UID: "ec94be10-a9ef-4259-b450-acd38f34d4e8"). InnerVolumeSpecName "kube-api-access-wsgm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:41.707003 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.706883 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-kube-api-access-s4b6r\" (UniqueName: \"kubernetes.io/projected/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-kube-api-access-s4b6r\") pod \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " Nov 02 22:04:41.707003 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.706939 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-home\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-home\") pod \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " Nov 02 22:04:41.707458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.707296 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-model-cache\") pod \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " Nov 02 22:04:41.707458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.707405 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tls-certs\" (UniqueName: \"kubernetes.io/secret/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tls-certs\") pod \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " Nov 02 22:04:41.708242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.707647 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-dshm\") pod \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " Nov 02 22:04:41.708242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.707687 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tmp-dir\") pod \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\" (UID: \"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f\") " Nov 02 22:04:41.708242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.708105 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.708242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.708126 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.708242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.708139 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94be10-a9ef-4259-b450-acd38f34d4e8-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.708242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.708152 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94be10-a9ef-4259-b450-acd38f34d4e8-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.708242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.708168 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgm4\" (UniqueName: \"kubernetes.io/projected/ec94be10-a9ef-4259-b450-acd38f34d4e8-kube-api-access-wsgm4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.708819 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.708724 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-model-cache" pod "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" (UID: "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.708819 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.708804 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-home" pod "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" (UID: "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.709080 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.709039 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tmp-dir" pod "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" (UID: "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.709439 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.709399 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-kube-api-access-s4b6r" pod "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" (UID: "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f"). InnerVolumeSpecName "kube-api-access-s4b6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:41.712931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.712905 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tls-certs" pod "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" (UID: "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:41.713166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.713005 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-dshm" pod "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" (UID: "522cb6cf-1fa2-4eee-8f06-f1e5fc55594f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.809686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.809652 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.809686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.809679 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.809686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.809690 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.810033 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.809699 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4b6r\" (UniqueName: \"kubernetes.io/projected/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-kube-api-access-s4b6r\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.810033 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.809710 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.810033 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:41.809718 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:42.040993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.040864 4441 generic.go:356] "Generic (PLEG): container finished" podID="4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" containerID="b9d0310f8bbe900ca8dfb67317feba294164d0b3637a46903aef0e5e25e5a898" exitCode=0 Nov 02 22:04:42.040993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.040955 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" event={"ID":"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26","Type":"ContainerDied","Data":"b9d0310f8bbe900ca8dfb67317feba294164d0b3637a46903aef0e5e25e5a898"} Nov 02 22:04:42.042697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.042673 4441 generic.go:356] "Generic (PLEG): container finished" podID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerID="e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06" exitCode=0 Nov 02 22:04:42.042837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.042717 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" Nov 02 22:04:42.042837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.042742 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" event={"ID":"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f","Type":"ContainerDied","Data":"e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06"} Nov 02 22:04:42.042837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.042769 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9" event={"ID":"522cb6cf-1fa2-4eee-8f06-f1e5fc55594f","Type":"ContainerDied","Data":"70417f99de0ca4c42b3aa89be2d610e6a47889654f9e9c647d52034eada09308"} Nov 02 22:04:42.042837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.042791 4441 scope.go:122] "RemoveContainer" containerID="e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06" Nov 02 22:04:42.044474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.044365 4441 generic.go:356] "Generic (PLEG): container finished" podID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerID="ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef" exitCode=0 Nov 02 22:04:42.044474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.044391 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" event={"ID":"ec94be10-a9ef-4259-b450-acd38f34d4e8","Type":"ContainerDied","Data":"ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef"} Nov 02 22:04:42.044474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.044412 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" Nov 02 22:04:42.044474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.044424 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc" event={"ID":"ec94be10-a9ef-4259-b450-acd38f34d4e8","Type":"ContainerDied","Data":"b7b2280b107d75db445a4c1693f90f60e04601e7c6b370ee1f772a770375508c"} Nov 02 22:04:42.046895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.046870 4441 generic.go:356] "Generic (PLEG): container finished" podID="3782096d-ec58-4a3c-9086-9ee05e10161e" containerID="3d89d086e4f8c6d197e7d092074b8598c706f2fa24dfaefd8d10accabd0f9a28" exitCode=0 Nov 02 22:04:42.047032 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.046955 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" event={"ID":"3782096d-ec58-4a3c-9086-9ee05e10161e","Type":"ContainerDied","Data":"3d89d086e4f8c6d197e7d092074b8598c706f2fa24dfaefd8d10accabd0f9a28"} Nov 02 22:04:42.056359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.056344 4441 scope.go:122] "RemoveContainer" containerID="e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06" Nov 02 22:04:42.056627 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:42.056609 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06\": container with ID starting with e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06 not found: ID does not exist" containerID="e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06" Nov 02 22:04:42.056715 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.056638 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06"} err="failed to get container status \"e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06\": rpc error: code = NotFound desc = could not find container \"e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06\": container with ID starting with e4fb277da87cd3de55cc656233390cc829c1c679807849e5845003eb11fc8e06 not found: ID does not exist" Nov 02 22:04:42.056715 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.056659 4441 scope.go:122] "RemoveContainer" containerID="ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef" Nov 02 22:04:42.066332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.066316 4441 scope.go:122] "RemoveContainer" containerID="ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef" Nov 02 22:04:42.066574 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:42.066557 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef\": container with ID starting with ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef not found: ID does not exist" containerID="ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef" Nov 02 22:04:42.066623 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.066582 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef"} err="failed to get container status \"ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef\": rpc error: code = NotFound desc = could not find container \"ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef\": container with ID starting with ae7254a6d54522d0d76adae1277eb204f8ebc607e87e8b418a326ba5491a01ef not found: ID does not exist" Nov 02 22:04:42.253831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.253786 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc"] Nov 02 22:04:42.257088 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.257060 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-2-w2-4d9ea0-kserve-774d8f7dff-2vjwc"] Nov 02 22:04:42.470957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.470926 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9"] Nov 02 22:04:42.475843 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.475814 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/premium-w2-4d9ea0-kserve-8674bf4b7c-nfxm9"] Nov 02 22:04:42.583174 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.583138 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" path="/var/lib/kubelet/pods/522cb6cf-1fa2-4eee-8f06-f1e5fc55594f/volumes" Nov 02 22:04:42.583656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:42.583635 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" path="/var/lib/kubelet/pods/ec94be10-a9ef-4259-b450-acd38f34d4e8/volumes" Nov 02 22:04:43.224335 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.224312 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:43.226354 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.226336 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:43.228330 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.228314 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-kube-api-access-rg9g4\" (UniqueName: \"kubernetes.io/projected/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-kube-api-access-rg9g4\") pod \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " Nov 02 22:04:43.228384 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.228373 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-openshift-ca\") pod \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " Nov 02 22:04:43.228500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.228478 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-xks-ca\") pod \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\" (UID: \"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26\") " Nov 02 22:04:43.228660 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.228643 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-openshift-ca" pod "4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" (UID: "4e6ddbb5-059a-4949-9d2e-5d08c29c8d26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:43.228852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.228826 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:43.230443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.230421 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-kube-api-access-rg9g4" pod "4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" (UID: "4e6ddbb5-059a-4949-9d2e-5d08c29c8d26"). InnerVolumeSpecName "kube-api-access-rg9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:43.230537 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.230421 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-xks-ca" pod "4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" (UID: "4e6ddbb5-059a-4949-9d2e-5d08c29c8d26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:43.329740 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.329664 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-openshift-ca\") pod \"3782096d-ec58-4a3c-9086-9ee05e10161e\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " Nov 02 22:04:43.329911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.329794 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-xks-ca\") pod \"3782096d-ec58-4a3c-9086-9ee05e10161e\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " Nov 02 22:04:43.329911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.329883 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3782096d-ec58-4a3c-9086-9ee05e10161e-kube-api-access-v2tsd\" (UniqueName: \"kubernetes.io/projected/3782096d-ec58-4a3c-9086-9ee05e10161e-kube-api-access-v2tsd\") pod \"3782096d-ec58-4a3c-9086-9ee05e10161e\" (UID: \"3782096d-ec58-4a3c-9086-9ee05e10161e\") " Nov 02 22:04:43.330248 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.330186 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-openshift-ca" pod "3782096d-ec58-4a3c-9086-9ee05e10161e" (UID: "3782096d-ec58-4a3c-9086-9ee05e10161e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:43.330646 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.330616 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:43.330646 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.330644 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:43.330806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.330669 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg9g4\" (UniqueName: \"kubernetes.io/projected/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26-kube-api-access-rg9g4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:43.332485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.332461 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-xks-ca" pod "3782096d-ec58-4a3c-9086-9ee05e10161e" (UID: "3782096d-ec58-4a3c-9086-9ee05e10161e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:43.333143 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.332736 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3782096d-ec58-4a3c-9086-9ee05e10161e-kube-api-access-v2tsd" pod "3782096d-ec58-4a3c-9086-9ee05e10161e" (UID: "3782096d-ec58-4a3c-9086-9ee05e10161e"). InnerVolumeSpecName "kube-api-access-v2tsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:43.421415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.421382 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" podUID="c7fd939e-a6b3-4579-b84d-c6b7da153315" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:43.431979 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.431949 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tsd\" (UniqueName: \"kubernetes.io/projected/3782096d-ec58-4a3c-9086-9ee05e10161e-kube-api-access-v2tsd\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:43.432085 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:43.431994 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3782096d-ec58-4a3c-9086-9ee05e10161e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:44.068403 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.068375 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" Nov 02 22:04:44.068605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.068414 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw" event={"ID":"4e6ddbb5-059a-4949-9d2e-5d08c29c8d26","Type":"ContainerDied","Data":"23a3729839677a976e743fbf1c4e9ffe2313871c4897e2b23bbce4060493cc08"} Nov 02 22:04:44.068605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.068463 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a3729839677a976e743fbf1c4e9ffe2313871c4897e2b23bbce4060493cc08" Nov 02 22:04:44.069771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.069749 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" Nov 02 22:04:44.069880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.069810 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t" event={"ID":"3782096d-ec58-4a3c-9086-9ee05e10161e","Type":"ContainerDied","Data":"5b6bc7b203a82658dc9bc2c65479a7932f384f91120a1dc813560f8fadd0d28e"} Nov 02 22:04:44.069880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.069849 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b6bc7b203a82658dc9bc2c65479a7932f384f91120a1dc813560f8fadd0d28e" Nov 02 22:04:44.845518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.845487 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k"] Nov 02 22:04:44.946137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:44.946106 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw"] Nov 02 22:04:45.100193 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:45.100118 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:45.217528 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:45.217493 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:45.345200 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:45.345170 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:47.466765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:47.466726 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" podUID="a9cffa72-fba9-446c-9780-130d7918e043" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:48.876623 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:48.876594 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 22:04:48.992861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:48.992768 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fd939e-a6b3-4579-b84d-c6b7da153315-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7fd939e-a6b3-4579-b84d-c6b7da153315-plugins-config-volume\") pod \"c7fd939e-a6b3-4579-b84d-c6b7da153315\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " Nov 02 22:04:48.993050 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:48.992928 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fd939e-a6b3-4579-b84d-c6b7da153315-kube-api-access-vvc72\" (UniqueName: \"kubernetes.io/projected/c7fd939e-a6b3-4579-b84d-c6b7da153315-kube-api-access-vvc72\") pod \"c7fd939e-a6b3-4579-b84d-c6b7da153315\" (UID: \"c7fd939e-a6b3-4579-b84d-c6b7da153315\") " Nov 02 22:04:48.993230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:48.993185 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fd939e-a6b3-4579-b84d-c6b7da153315-plugins-config-volume" pod "c7fd939e-a6b3-4579-b84d-c6b7da153315" (UID: "c7fd939e-a6b3-4579-b84d-c6b7da153315"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:48.993359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:48.993336 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7fd939e-a6b3-4579-b84d-c6b7da153315-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:48.995037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:48.995012 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fd939e-a6b3-4579-b84d-c6b7da153315-kube-api-access-vvc72" pod "c7fd939e-a6b3-4579-b84d-c6b7da153315" (UID: "c7fd939e-a6b3-4579-b84d-c6b7da153315"). InnerVolumeSpecName "kube-api-access-vvc72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:49.094466 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.094434 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvc72\" (UniqueName: \"kubernetes.io/projected/c7fd939e-a6b3-4579-b84d-c6b7da153315-kube-api-access-vvc72\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:49.111227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.111196 4441 generic.go:356] "Generic (PLEG): container finished" podID="c7fd939e-a6b3-4579-b84d-c6b7da153315" containerID="b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148" exitCode=0 Nov 02 22:04:49.111355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.111241 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" Nov 02 22:04:49.111355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.111271 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" event={"ID":"c7fd939e-a6b3-4579-b84d-c6b7da153315","Type":"ContainerDied","Data":"b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148"} Nov 02 22:04:49.111355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.111333 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq" event={"ID":"c7fd939e-a6b3-4579-b84d-c6b7da153315","Type":"ContainerDied","Data":"af8e00fa3d1ee75078a38060d830dcd0a502e9de56c1f695358ebb5972f9b673"} Nov 02 22:04:49.111487 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.111358 4441 scope.go:122] "RemoveContainer" containerID="b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148" Nov 02 22:04:49.121662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.121641 4441 scope.go:122] "RemoveContainer" containerID="b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148" Nov 02 22:04:49.121943 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:49.121924 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148\": container with ID starting with b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148 not found: ID does not exist" containerID="b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148" Nov 02 22:04:49.122038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.121950 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148"} err="failed to get container status \"b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148\": rpc error: code = NotFound desc = could not find container \"b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148\": container with ID starting with b564461c730fcf46059469b7e46e073007484f6c3703d405b05043288f0e3148 not found: ID does not exist" Nov 02 22:04:49.267802 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.267716 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq"] Nov 02 22:04:49.271650 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.271624 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9a6cf2-7d98fcf679-zzqnq"] Nov 02 22:04:49.587156 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.587128 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:49.846536 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.846455 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" podUID="e9e9548b-d8f1-497a-94e6-1d6b72c8a571" containerName="payload-processing" containerID="cri-o://3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2" gracePeriod=25 Nov 02 22:04:49.946515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:49.946480 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" podUID="f6794621-aebc-4aac-9deb-e2e634c09275" containerName="payload-pre-processing" containerID="cri-o://c56e2cf8fc9879a5cee04eb0a49e6b4f60a892560fa3b3765e61d3bf04b0c521" gracePeriod=25 Nov 02 22:04:50.106081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.106020 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 22:04:50.124348 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.124316 4441 generic.go:356] "Generic (PLEG): container finished" podID="e9e9548b-d8f1-497a-94e6-1d6b72c8a571" containerID="3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2" exitCode=0 Nov 02 22:04:50.124544 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.124367 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" Nov 02 22:04:50.124544 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.124505 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" event={"ID":"e9e9548b-d8f1-497a-94e6-1d6b72c8a571","Type":"ContainerDied","Data":"3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2"} Nov 02 22:04:50.124710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.124543 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k" event={"ID":"e9e9548b-d8f1-497a-94e6-1d6b72c8a571","Type":"ContainerDied","Data":"505a29ea0fb3df9c048965f9b56cc21f508bdccf00674c9d9c0483b78c8db751"} Nov 02 22:04:50.124710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.124579 4441 scope.go:122] "RemoveContainer" containerID="3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2" Nov 02 22:04:50.128244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.128087 4441 generic.go:356] "Generic (PLEG): container finished" podID="f6794621-aebc-4aac-9deb-e2e634c09275" containerID="c56e2cf8fc9879a5cee04eb0a49e6b4f60a892560fa3b3765e61d3bf04b0c521" exitCode=0 Nov 02 22:04:50.128244 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.128180 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" event={"ID":"f6794621-aebc-4aac-9deb-e2e634c09275","Type":"ContainerDied","Data":"c56e2cf8fc9879a5cee04eb0a49e6b4f60a892560fa3b3765e61d3bf04b0c521"} Nov 02 22:04:50.140077 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.139987 4441 scope.go:122] "RemoveContainer" containerID="3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2" Nov 02 22:04:50.140327 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:50.140299 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2\": container with ID starting with 3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2 not found: ID does not exist" containerID="3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2" Nov 02 22:04:50.140426 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.140336 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2"} err="failed to get container status \"3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2\": rpc error: code = NotFound desc = could not find container \"3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2\": container with ID starting with 3e523ab00e0cdc514e6cfe531129dd082a0dd14cd9312490d1888578c0ce3ac2 not found: ID does not exist" Nov 02 22:04:50.200864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.200842 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 22:04:50.203885 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.203864 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-kube-api-access-854hr\" (UniqueName: \"kubernetes.io/projected/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-kube-api-access-854hr\") pod \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " Nov 02 22:04:50.203989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.203904 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-plugins-config-volume\") pod \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\" (UID: \"e9e9548b-d8f1-497a-94e6-1d6b72c8a571\") " Nov 02 22:04:50.204259 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.204235 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-plugins-config-volume" pod "e9e9548b-d8f1-497a-94e6-1d6b72c8a571" (UID: "e9e9548b-d8f1-497a-94e6-1d6b72c8a571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:50.205795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.205778 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-kube-api-access-854hr" pod "e9e9548b-d8f1-497a-94e6-1d6b72c8a571" (UID: "e9e9548b-d8f1-497a-94e6-1d6b72c8a571"). InnerVolumeSpecName "kube-api-access-854hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:50.304926 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.304899 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6794621-aebc-4aac-9deb-e2e634c09275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6794621-aebc-4aac-9deb-e2e634c09275-plugins-config-volume\") pod \"f6794621-aebc-4aac-9deb-e2e634c09275\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " Nov 02 22:04:50.305114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.305006 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6794621-aebc-4aac-9deb-e2e634c09275-kube-api-access-kxhth\" (UniqueName: \"kubernetes.io/projected/f6794621-aebc-4aac-9deb-e2e634c09275-kube-api-access-kxhth\") pod \"f6794621-aebc-4aac-9deb-e2e634c09275\" (UID: \"f6794621-aebc-4aac-9deb-e2e634c09275\") " Nov 02 22:04:50.305276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.305252 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6794621-aebc-4aac-9deb-e2e634c09275-plugins-config-volume" pod "f6794621-aebc-4aac-9deb-e2e634c09275" (UID: "f6794621-aebc-4aac-9deb-e2e634c09275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:50.305357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.305331 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854hr\" (UniqueName: \"kubernetes.io/projected/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-kube-api-access-854hr\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:50.305357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.305349 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e9548b-d8f1-497a-94e6-1d6b72c8a571-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:50.305434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.305359 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6794621-aebc-4aac-9deb-e2e634c09275-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:50.307022 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.306987 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6794621-aebc-4aac-9deb-e2e634c09275-kube-api-access-kxhth" pod "f6794621-aebc-4aac-9deb-e2e634c09275" (UID: "f6794621-aebc-4aac-9deb-e2e634c09275"). InnerVolumeSpecName "kube-api-access-kxhth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:50.406837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.406760 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhth\" (UniqueName: \"kubernetes.io/projected/f6794621-aebc-4aac-9deb-e2e634c09275-kube-api-access-kxhth\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:50.539684 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.539527 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k"] Nov 02 22:04:50.542741 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.542718 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e2b27-5569b97fc9-q286k"] Nov 02 22:04:50.583107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.583080 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fd939e-a6b3-4579-b84d-c6b7da153315" path="/var/lib/kubelet/pods/c7fd939e-a6b3-4579-b84d-c6b7da153315/volumes" Nov 02 22:04:50.583453 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:50.583441 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e9548b-d8f1-497a-94e6-1d6b72c8a571" path="/var/lib/kubelet/pods/e9e9548b-d8f1-497a-94e6-1d6b72c8a571/volumes" Nov 02 22:04:51.142139 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:51.142071 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" Nov 02 22:04:51.142139 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:51.142091 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw" event={"ID":"f6794621-aebc-4aac-9deb-e2e634c09275","Type":"ContainerDied","Data":"79ae15f09fd06a84f84f94e4035fcab9280090a30f56ddcee2702f43ea157340"} Nov 02 22:04:51.142626 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:51.142178 4441 scope.go:122] "RemoveContainer" containerID="c56e2cf8fc9879a5cee04eb0a49e6b4f60a892560fa3b3765e61d3bf04b0c521" Nov 02 22:04:51.297878 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:51.297837 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw"] Nov 02 22:04:51.317188 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:51.317150 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e2b27-987d59d59-4j6sw"] Nov 02 22:04:51.414859 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:51.414780 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:52.391943 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:52.391910 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:52.538907 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:52.538876 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:52.583279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:52.583247 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6794621-aebc-4aac-9deb-e2e634c09275" path="/var/lib/kubelet/pods/f6794621-aebc-4aac-9deb-e2e634c09275/volumes" Nov 02 22:04:52.971864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:52.971821 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" containerID="cri-o://c714d4d170d086058ed71dfb45aec03e92592384c7d8867773f02ccce109448e" gracePeriod=45 Nov 02 22:04:53.160218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.160188 4441 generic.go:356] "Generic (PLEG): container finished" podID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerID="c714d4d170d086058ed71dfb45aec03e92592384c7d8867773f02ccce109448e" exitCode=0 Nov 02 22:04:53.160355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.160262 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" event={"ID":"f9377f9b-5bcc-4347-a55d-8b38059f19ca","Type":"ContainerDied","Data":"c714d4d170d086058ed71dfb45aec03e92592384c7d8867773f02ccce109448e"} Nov 02 22:04:53.226036 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.225979 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 22:04:53.335313 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335284 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-dshm\") pod \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " Nov 02 22:04:53.335464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335398 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9377f9b-5bcc-4347-a55d-8b38059f19ca-kube-api-access-m5s7r\" (UniqueName: \"kubernetes.io/projected/f9377f9b-5bcc-4347-a55d-8b38059f19ca-kube-api-access-m5s7r\") pod \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " Nov 02 22:04:53.335464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335428 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-home\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-home\") pod \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " Nov 02 22:04:53.335464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335445 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-model-cache\") pod \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " Nov 02 22:04:53.335642 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335472 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tmp-dir\") pod \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " Nov 02 22:04:53.335698 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335677 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tls-certs\") pod \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\" (UID: \"f9377f9b-5bcc-4347-a55d-8b38059f19ca\") " Nov 02 22:04:53.335760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335710 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-home" pod "f9377f9b-5bcc-4347-a55d-8b38059f19ca" (UID: "f9377f9b-5bcc-4347-a55d-8b38059f19ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.335760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.335722 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tmp-dir" pod "f9377f9b-5bcc-4347-a55d-8b38059f19ca" (UID: "f9377f9b-5bcc-4347-a55d-8b38059f19ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.336082 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.336040 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-model-cache" pod "f9377f9b-5bcc-4347-a55d-8b38059f19ca" (UID: "f9377f9b-5bcc-4347-a55d-8b38059f19ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.336182 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.336101 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.336182 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.336118 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.337897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.337874 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-dshm" pod "f9377f9b-5bcc-4347-a55d-8b38059f19ca" (UID: "f9377f9b-5bcc-4347-a55d-8b38059f19ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.338250 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.338223 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9377f9b-5bcc-4347-a55d-8b38059f19ca-kube-api-access-m5s7r" pod "f9377f9b-5bcc-4347-a55d-8b38059f19ca" (UID: "f9377f9b-5bcc-4347-a55d-8b38059f19ca"). InnerVolumeSpecName "kube-api-access-m5s7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:53.338443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.338421 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tls-certs" pod "f9377f9b-5bcc-4347-a55d-8b38059f19ca" (UID: "f9377f9b-5bcc-4347-a55d-8b38059f19ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:53.436895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.436863 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5s7r\" (UniqueName: \"kubernetes.io/projected/f9377f9b-5bcc-4347-a55d-8b38059f19ca-kube-api-access-m5s7r\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.436895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.436891 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.436895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.436902 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9377f9b-5bcc-4347-a55d-8b38059f19ca-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.437408 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.436911 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9377f9b-5bcc-4347-a55d-8b38059f19ca-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.623013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.622962 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62"] Nov 02 22:04:53.623502 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623484 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e9e9548b-d8f1-497a-94e6-1d6b72c8a571" containerName="payload-processing" Nov 02 22:04:53.623502 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623502 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e9548b-d8f1-497a-94e6-1d6b72c8a571" containerName="payload-processing" Nov 02 22:04:53.623613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623522 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" Nov 02 22:04:53.623613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623534 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" Nov 02 22:04:53.623613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623556 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f6794621-aebc-4aac-9deb-e2e634c09275" containerName="payload-pre-processing" Nov 02 22:04:53.623613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623563 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6794621-aebc-4aac-9deb-e2e634c09275" containerName="payload-pre-processing" Nov 02 22:04:53.623613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623597 4441 cpu_manager.go:414] "containerMap: removing container" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" Nov 02 22:04:53.623613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623602 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623629 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623640 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623649 4441 cpu_manager.go:414] "containerMap: removing container" podUID="c7fd939e-a6b3-4579-b84d-c6b7da153315" containerName="payload-pre-processing" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623653 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fd939e-a6b3-4579-b84d-c6b7da153315" containerName="payload-pre-processing" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623661 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" containerName="revoke-keys" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623666 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" containerName="revoke-keys" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623674 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3782096d-ec58-4a3c-9086-9ee05e10161e" containerName="revoke-keys" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623679 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3782096d-ec58-4a3c-9086-9ee05e10161e" containerName="revoke-keys" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623748 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6794621-aebc-4aac-9deb-e2e634c09275" containerName="payload-pre-processing" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623757 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="522cb6cf-1fa2-4eee-8f06-f1e5fc55594f" containerName="main" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623765 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" containerName="main" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623771 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fd939e-a6b3-4579-b84d-c6b7da153315" containerName="payload-pre-processing" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623780 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" containerName="revoke-keys" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623786 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3782096d-ec58-4a3c-9086-9ee05e10161e" containerName="revoke-keys" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623792 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e9548b-d8f1-497a-94e6-1d6b72c8a571" containerName="payload-processing" Nov 02 22:04:53.623838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.623798 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec94be10-a9ef-4259-b450-acd38f34d4e8" containerName="main" Nov 02 22:04:53.627222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.627202 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.677074 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.677038 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m"] Nov 02 22:04:53.681112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.681090 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.681302 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.681281 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62"] Nov 02 22:04:53.720307 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.720280 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:04:53.720307 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.720285 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:04:53.720496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.720328 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:04:53.723112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.723093 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:04:53.740376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.739899 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.740376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.740022 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.740376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.740053 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.740376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.740150 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6chv\" (UniqueName: \"kubernetes.io/projected/fab1d0b4-7423-4644-b4e6-21dedcd645ca-kube-api-access-w6chv\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.740376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.740205 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.740376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.740231 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnt9\" (UniqueName: \"kubernetes.io/projected/0d746677-71a0-4ebb-ba09-b45fa261e8d6-kube-api-access-gcnt9\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.745920 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.745896 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m"] Nov 02 22:04:53.805836 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.805802 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:53.841714 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.841683 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6chv\" (UniqueName: \"kubernetes.io/projected/fab1d0b4-7423-4644-b4e6-21dedcd645ca-kube-api-access-w6chv\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.841878 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.841755 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.841878 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.841787 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnt9\" (UniqueName: \"kubernetes.io/projected/0d746677-71a0-4ebb-ba09-b45fa261e8d6-kube-api-access-gcnt9\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.841878 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.841830 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.842107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.841898 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.842107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.841926 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.842730 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.842706 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.842841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.842730 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.844343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.844317 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.844441 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.844373 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.895385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.895335 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnt9\" (UniqueName: \"kubernetes.io/projected/0d746677-71a0-4ebb-ba09-b45fa261e8d6-kube-api-access-gcnt9\") pod \"maas-api-revoke-sub-fa7f859c-mkm62\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.895549 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.895528 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6chv\" (UniqueName: \"kubernetes.io/projected/fab1d0b4-7423-4644-b4e6-21dedcd645ca-kube-api-access-w6chv\") pod \"maas-api-revoke-sub-773e8060-g284m\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:53.941250 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.941219 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:53.997165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:53.997112 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:54.168905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.168817 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" Nov 02 22:04:54.168905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.168851 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q" event={"ID":"f9377f9b-5bcc-4347-a55d-8b38059f19ca","Type":"ContainerDied","Data":"61966cd8f87e378b14d33e5f0da1eef00238bb43ce49a7a34626d982d2fa8d11"} Nov 02 22:04:54.168905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.168901 4441 scope.go:122] "RemoveContainer" containerID="c714d4d170d086058ed71dfb45aec03e92592384c7d8867773f02ccce109448e" Nov 02 22:04:54.311939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.311899 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62"] Nov 02 22:04:54.312946 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:54.312913 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d746677_71a0_4ebb_ba09_b45fa261e8d6.slice/crio-8b5c91bc7d3c91e1245628c50a6d0766998e15fe29e9dbe727e90354dbe43138 WatchSource:0}: Error finding container 8b5c91bc7d3c91e1245628c50a6d0766998e15fe29e9dbe727e90354dbe43138: Status 404 returned error can't find the container with id 8b5c91bc7d3c91e1245628c50a6d0766998e15fe29e9dbe727e90354dbe43138 Nov 02 22:04:54.325535 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.325510 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q"] Nov 02 22:04:54.329558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.329539 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/simulator-w4-4e2b27-kserve-7b95c8f9bc-l452q"] Nov 02 22:04:54.412622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.412601 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m"] Nov 02 22:04:54.414132 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:04:54.414104 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab1d0b4_7423_4644_b4e6_21dedcd645ca.slice/crio-f3af5e59835c263ad0a320624366aad0c5f1a3d496534955dc4218ba81c52aef WatchSource:0}: Error finding container f3af5e59835c263ad0a320624366aad0c5f1a3d496534955dc4218ba81c52aef: Status 404 returned error can't find the container with id f3af5e59835c263ad0a320624366aad0c5f1a3d496534955dc4218ba81c52aef Nov 02 22:04:54.584767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.584741 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9377f9b-5bcc-4347-a55d-8b38059f19ca" path="/var/lib/kubelet/pods/f9377f9b-5bcc-4347-a55d-8b38059f19ca/volumes" Nov 02 22:04:54.960494 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.960418 4441 scope.go:122] "RemoveContainer" containerID="f58c92d1868244a41c8441b6dbbbe84898e7e6341d8a3567ca9d39fe6e6a3cf5" Nov 02 22:04:54.970368 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:54.970344 4441 scope.go:122] "RemoveContainer" containerID="36a8b2a9000a4f14cf2a7b44a5ea4f2ff7d73f073716adb7e5231ef14d3e2985" Nov 02 22:04:55.177462 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.177431 4441 generic.go:356] "Generic (PLEG): container finished" podID="0d746677-71a0-4ebb-ba09-b45fa261e8d6" containerID="ee1cd48fa809c5e4c9a286b276d121dd8fe627f544501c0a5a67107d8df35bc4" exitCode=0 Nov 02 22:04:55.177596 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.177520 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" event={"ID":"0d746677-71a0-4ebb-ba09-b45fa261e8d6","Type":"ContainerDied","Data":"ee1cd48fa809c5e4c9a286b276d121dd8fe627f544501c0a5a67107d8df35bc4"} Nov 02 22:04:55.177596 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.177577 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" event={"ID":"0d746677-71a0-4ebb-ba09-b45fa261e8d6","Type":"ContainerStarted","Data":"8b5c91bc7d3c91e1245628c50a6d0766998e15fe29e9dbe727e90354dbe43138"} Nov 02 22:04:55.179208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.179184 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" event={"ID":"fab1d0b4-7423-4644-b4e6-21dedcd645ca","Type":"ContainerStarted","Data":"a1acbe8f8013c52c0b173594be7bac44a4015e082b342149cbf8532301474595"} Nov 02 22:04:55.179296 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.179226 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" event={"ID":"fab1d0b4-7423-4644-b4e6-21dedcd645ca","Type":"ContainerStarted","Data":"f3af5e59835c263ad0a320624366aad0c5f1a3d496534955dc4218ba81c52aef"} Nov 02 22:04:55.275876 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.275820 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:55.393314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.393272 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" podStartSLOduration=2.393259442 podStartE2EDuration="2.393259442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:55.390550332 +0000 UTC m=+3124.946178696" watchObservedRunningTime="2026-11-02 22:04:55.393259442 +0000 UTC m=+3124.948887819" Nov 02 22:04:55.509176 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.509148 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:55.646410 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.646376 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:55.751590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.751557 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv"] Nov 02 22:04:55.754859 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:55.754837 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9a6cf2-6r2qv"] Nov 02 22:04:56.173269 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.173221 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:56.188292 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.188259 4441 generic.go:356] "Generic (PLEG): container finished" podID="fab1d0b4-7423-4644-b4e6-21dedcd645ca" containerID="a1acbe8f8013c52c0b173594be7bac44a4015e082b342149cbf8532301474595" exitCode=0 Nov 02 22:04:56.188466 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.188303 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" event={"ID":"fab1d0b4-7423-4644-b4e6-21dedcd645ca","Type":"ContainerDied","Data":"a1acbe8f8013c52c0b173594be7bac44a4015e082b342149cbf8532301474595"} Nov 02 22:04:56.334201 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.334167 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:56.466325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.466240 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-xks-ca\") pod \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " Nov 02 22:04:56.466503 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.466329 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-openshift-ca\") pod \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " Nov 02 22:04:56.466503 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.466358 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d746677-71a0-4ebb-ba09-b45fa261e8d6-kube-api-access-gcnt9\" (UniqueName: \"kubernetes.io/projected/0d746677-71a0-4ebb-ba09-b45fa261e8d6-kube-api-access-gcnt9\") pod \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\" (UID: \"0d746677-71a0-4ebb-ba09-b45fa261e8d6\") " Nov 02 22:04:56.466754 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.466729 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-openshift-ca" pod "0d746677-71a0-4ebb-ba09-b45fa261e8d6" (UID: "0d746677-71a0-4ebb-ba09-b45fa261e8d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:56.468405 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.468378 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-xks-ca" pod "0d746677-71a0-4ebb-ba09-b45fa261e8d6" (UID: "0d746677-71a0-4ebb-ba09-b45fa261e8d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:56.468533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.468499 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d746677-71a0-4ebb-ba09-b45fa261e8d6-kube-api-access-gcnt9" pod "0d746677-71a0-4ebb-ba09-b45fa261e8d6" (UID: "0d746677-71a0-4ebb-ba09-b45fa261e8d6"). InnerVolumeSpecName "kube-api-access-gcnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:56.567734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.567701 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:56.567734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.567730 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcnt9\" (UniqueName: \"kubernetes.io/projected/0d746677-71a0-4ebb-ba09-b45fa261e8d6-kube-api-access-gcnt9\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:56.567734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.567741 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d746677-71a0-4ebb-ba09-b45fa261e8d6-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:56.582764 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:56.582734 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c654ce1d-3199-41c4-a4a1-32e6cde79fc6" path="/var/lib/kubelet/pods/c654ce1d-3199-41c4-a4a1-32e6cde79fc6/volumes" Nov 02 22:04:57.196872 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.196842 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" Nov 02 22:04:57.197370 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.196878 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62" event={"ID":"0d746677-71a0-4ebb-ba09-b45fa261e8d6","Type":"ContainerDied","Data":"8b5c91bc7d3c91e1245628c50a6d0766998e15fe29e9dbe727e90354dbe43138"} Nov 02 22:04:57.197370 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.196938 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b5c91bc7d3c91e1245628c50a6d0766998e15fe29e9dbe727e90354dbe43138" Nov 02 22:04:57.324220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.324191 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:57.373944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.373917 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-xks-ca\") pod \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " Nov 02 22:04:57.374090 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.374055 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-openshift-ca\") pod \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " Nov 02 22:04:57.374153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.374088 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab1d0b4-7423-4644-b4e6-21dedcd645ca-kube-api-access-w6chv\" (UniqueName: \"kubernetes.io/projected/fab1d0b4-7423-4644-b4e6-21dedcd645ca-kube-api-access-w6chv\") pod \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\" (UID: \"fab1d0b4-7423-4644-b4e6-21dedcd645ca\") " Nov 02 22:04:57.374401 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.374380 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-openshift-ca" pod "fab1d0b4-7423-4644-b4e6-21dedcd645ca" (UID: "fab1d0b4-7423-4644-b4e6-21dedcd645ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:57.375920 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.375901 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-xks-ca" pod "fab1d0b4-7423-4644-b4e6-21dedcd645ca" (UID: "fab1d0b4-7423-4644-b4e6-21dedcd645ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:57.376041 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.376025 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab1d0b4-7423-4644-b4e6-21dedcd645ca-kube-api-access-w6chv" pod "fab1d0b4-7423-4644-b4e6-21dedcd645ca" (UID: "fab1d0b4-7423-4644-b4e6-21dedcd645ca"). InnerVolumeSpecName "kube-api-access-w6chv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:57.466327 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.466240 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" podUID="a9cffa72-fba9-446c-9780-130d7918e043" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:57.475266 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.475235 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:57.475266 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.475262 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab1d0b4-7423-4644-b4e6-21dedcd645ca-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:57.475435 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:57.475279 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6chv\" (UniqueName: \"kubernetes.io/projected/fab1d0b4-7423-4644-b4e6-21dedcd645ca-kube-api-access-w6chv\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:58.032502 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.032467 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:58.147825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.147794 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:58.206580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.206341 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" Nov 02 22:04:58.206580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.206358 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m" event={"ID":"fab1d0b4-7423-4644-b4e6-21dedcd645ca","Type":"ContainerDied","Data":"f3af5e59835c263ad0a320624366aad0c5f1a3d496534955dc4218ba81c52aef"} Nov 02 22:04:58.206580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.206418 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3af5e59835c263ad0a320624366aad0c5f1a3d496534955dc4218ba81c52aef" Nov 02 22:04:58.445742 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.445719 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 22:04:58.587551 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.587457 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cffa72-fba9-446c-9780-130d7918e043-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cffa72-fba9-446c-9780-130d7918e043-plugins-config-volume\") pod \"a9cffa72-fba9-446c-9780-130d7918e043\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " Nov 02 22:04:58.587732 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.587616 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cffa72-fba9-446c-9780-130d7918e043-kube-api-access-9qm6s\" (UniqueName: \"kubernetes.io/projected/a9cffa72-fba9-446c-9780-130d7918e043-kube-api-access-9qm6s\") pod \"a9cffa72-fba9-446c-9780-130d7918e043\" (UID: \"a9cffa72-fba9-446c-9780-130d7918e043\") " Nov 02 22:04:58.588138 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.588109 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cffa72-fba9-446c-9780-130d7918e043-plugins-config-volume" pod "a9cffa72-fba9-446c-9780-130d7918e043" (UID: "a9cffa72-fba9-446c-9780-130d7918e043"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:58.590077 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.590049 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cffa72-fba9-446c-9780-130d7918e043-kube-api-access-9qm6s" pod "a9cffa72-fba9-446c-9780-130d7918e043" (UID: "a9cffa72-fba9-446c-9780-130d7918e043"). InnerVolumeSpecName "kube-api-access-9qm6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:58.689132 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.689104 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qm6s\" (UniqueName: \"kubernetes.io/projected/a9cffa72-fba9-446c-9780-130d7918e043-kube-api-access-9qm6s\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:58.689132 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:58.689126 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cffa72-fba9-446c-9780-130d7918e043-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:59.216341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.216306 4441 generic.go:356] "Generic (PLEG): container finished" podID="a9cffa72-fba9-446c-9780-130d7918e043" containerID="e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c" exitCode=0 Nov 02 22:04:59.216846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.216367 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" Nov 02 22:04:59.216846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.216389 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" event={"ID":"a9cffa72-fba9-446c-9780-130d7918e043","Type":"ContainerDied","Data":"e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c"} Nov 02 22:04:59.216846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.216443 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg" event={"ID":"a9cffa72-fba9-446c-9780-130d7918e043","Type":"ContainerDied","Data":"6146bc32dfbd1114e3f92beb8eb03b00942c55ba38b1c3982c2502b6e9d3235d"} Nov 02 22:04:59.216846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.216466 4441 scope.go:122] "RemoveContainer" containerID="e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c" Nov 02 22:04:59.226576 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.226560 4441 scope.go:122] "RemoveContainer" containerID="e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c" Nov 02 22:04:59.226851 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:04:59.226829 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c\": container with ID starting with e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c not found: ID does not exist" containerID="e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c" Nov 02 22:04:59.226897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.226860 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c"} err="failed to get container status \"e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c\": rpc error: code = NotFound desc = could not find container \"e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c\": container with ID starting with e73dcb4c5e2b548264d52f6dc0ba181f73a2ea0ce0bff3e1e75284ce5e8a580c not found: ID does not exist" Nov 02 22:04:59.373250 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.373213 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg"] Nov 02 22:04:59.422283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.422249 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4d9ea0-7c6896dc8c-d9bwg"] Nov 02 22:04:59.616614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.616574 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk"] Nov 02 22:04:59.617079 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617059 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a9cffa72-fba9-446c-9780-130d7918e043" containerName="payload-pre-processing" Nov 02 22:04:59.617079 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617079 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cffa72-fba9-446c-9780-130d7918e043" containerName="payload-pre-processing" Nov 02 22:04:59.617240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617096 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0d746677-71a0-4ebb-ba09-b45fa261e8d6" containerName="revoke-keys" Nov 02 22:04:59.617240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617101 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d746677-71a0-4ebb-ba09-b45fa261e8d6" containerName="revoke-keys" Nov 02 22:04:59.617240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617119 4441 cpu_manager.go:414] "containerMap: removing container" podUID="fab1d0b4-7423-4644-b4e6-21dedcd645ca" containerName="revoke-keys" Nov 02 22:04:59.617240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617124 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab1d0b4-7423-4644-b4e6-21dedcd645ca" containerName="revoke-keys" Nov 02 22:04:59.617240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617199 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab1d0b4-7423-4644-b4e6-21dedcd645ca" containerName="revoke-keys" Nov 02 22:04:59.617240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617218 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cffa72-fba9-446c-9780-130d7918e043" containerName="payload-pre-processing" Nov 02 22:04:59.617240 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.617228 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d746677-71a0-4ebb-ba09-b45fa261e8d6" containerName="revoke-keys" Nov 02 22:04:59.621760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.621740 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.671533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.671473 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49c06d\"/\"kube-root-ca.crt\"" Nov 02 22:04:59.672135 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.672115 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49c06d\"/\"openshift-service-ca.crt\"" Nov 02 22:04:59.672725 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.672695 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49c06d\"/\"test-model-49c06d-kserve-self-signed-certs\"" Nov 02 22:04:59.672841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.672779 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49c06d\"/\"default-dockercfg-t2b2f\"" Nov 02 22:04:59.676082 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.676058 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk"] Nov 02 22:04:59.694553 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.694528 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:59.698733 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.698714 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbef99f8-743b-437c-9866-8bab0cf3c788-tls-certs\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.698803 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.698793 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-model-cache\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.698889 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.698867 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjjw\" (UniqueName: \"kubernetes.io/projected/cbef99f8-743b-437c-9866-8bab0cf3c788-kube-api-access-2wjjw\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.698889 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.698888 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-tmp-dir\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.699099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.698932 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-home\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.699099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.698998 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-dshm\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800195 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800160 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-model-cache\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800208 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjjw\" (UniqueName: \"kubernetes.io/projected/cbef99f8-743b-437c-9866-8bab0cf3c788-kube-api-access-2wjjw\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800237 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-tmp-dir\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800288 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-home\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800320 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-dshm\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800368 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbef99f8-743b-437c-9866-8bab0cf3c788-tls-certs\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800709 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800684 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-model-cache\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800717 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-tmp-dir\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.800825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.800763 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-home\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.802722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.802700 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-dshm\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.802910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.802892 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbef99f8-743b-437c-9866-8bab0cf3c788-tls-certs\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.854172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.854145 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjjw\" (UniqueName: \"kubernetes.io/projected/cbef99f8-743b-437c-9866-8bab0cf3c788-kube-api-access-2wjjw\") pod \"test-model-49c06d-kserve-58f654bfd7-tfxnk\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:04:59.932835 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:04:59.932777 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:05:00.304190 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:05:00.304158 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbef99f8_743b_437c_9866_8bab0cf3c788.slice/crio-ae0d8270827f1eb374fe9bee7b24b4931e88a80ac8e03b1c196005c5c3ee2fcb WatchSource:0}: Error finding container ae0d8270827f1eb374fe9bee7b24b4931e88a80ac8e03b1c196005c5c3ee2fcb: Status 404 returned error can't find the container with id ae0d8270827f1eb374fe9bee7b24b4931e88a80ac8e03b1c196005c5c3ee2fcb Nov 02 22:05:00.304622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:00.304349 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk"] Nov 02 22:05:00.389592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:00.389566 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:00.584325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:00.584288 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cffa72-fba9-446c-9780-130d7918e043" path="/var/lib/kubelet/pods/a9cffa72-fba9-446c-9780-130d7918e043/volumes" Nov 02 22:05:01.234992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:01.234934 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" event={"ID":"cbef99f8-743b-437c-9866-8bab0cf3c788","Type":"ContainerStarted","Data":"da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916"} Nov 02 22:05:01.235157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:01.234997 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" event={"ID":"cbef99f8-743b-437c-9866-8bab0cf3c788","Type":"ContainerStarted","Data":"ae0d8270827f1eb374fe9bee7b24b4931e88a80ac8e03b1c196005c5c3ee2fcb"} Nov 02 22:05:01.344850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:01.344791 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" podStartSLOduration=2.344775541 podStartE2EDuration="2.344775541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:01.343129775 +0000 UTC m=+3130.898758114" watchObservedRunningTime="2026-11-02 22:05:01.344775541 +0000 UTC m=+3130.900403877" Nov 02 22:05:01.933131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:01.933099 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:05:02.015614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:02.015580 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:02.141112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:02.141073 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:02.349073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:02.349039 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:03.398839 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:03.398800 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:03.608756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:03.608720 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:04.449961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:04.449918 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:04.590392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:04.590347 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:05.117595 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:05.117562 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:05.249809 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:05.249774 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:05.953102 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:05.953065 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:06.181437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:06.181398 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:06.306212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:06.306121 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:06.460881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:06.460846 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt"] Nov 02 22:05:06.502720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:06.502668 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4d9ea0-nsrjt"] Nov 02 22:05:06.583831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:06.583795 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d2822f-3714-4402-a11b-2798ebb41753" path="/var/lib/kubelet/pods/a3d2822f-3714-4402-a11b-2798ebb41753/volumes" Nov 02 22:05:06.808057 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:06.808023 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:09.933576 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:09.933540 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:05:09.946146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:09.946118 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:05:10.327792 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:10.327763 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:05:10.616062 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:05:10.615958 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:05:11.101983 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:11.101940 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:11.479386 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:11.479301 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:13.295255 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:13.295223 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:13.414381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:13.414354 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:13.616569 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:13.616533 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:14.625475 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:14.625445 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:14.749270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:14.749238 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:15.099618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:15.099588 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:15.513840 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:15.513760 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:16.574716 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:16.574683 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:16.793350 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:16.793314 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:16.919307 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:16.919230 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:17.044864 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:17.044833 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:17.050119 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:17.050092 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv"] Nov 02 22:05:17.097160 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:17.097132 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e2b27-lxlbv"] Nov 02 22:05:17.433339 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:17.433307 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:18.585861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:18.585824 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768bfa6e-d313-466d-a13c-66ab8ef2de80" path="/var/lib/kubelet/pods/768bfa6e-d313-466d-a13c-66ab8ef2de80/volumes" Nov 02 22:05:20.919452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:20.919422 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:21.674629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:21.674595 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:23.215401 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:23.215352 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:23.344843 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:23.344811 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:23.544603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:23.544514 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:27.569799 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:27.569726 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:30.684857 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:30.684820 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:35.202586 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:35.202545 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:39.683465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:39.683434 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:40.044962 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:40.044874 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm"] Nov 02 22:05:40.091530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:40.091501 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04df275-gp9tm"] Nov 02 22:05:40.583447 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:40.583407 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed73a45b-72f3-4deb-91e1-6fa8490f0df8" path="/var/lib/kubelet/pods/ed73a45b-72f3-4deb-91e1-6fa8490f0df8/volumes" Nov 02 22:05:40.616825 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:05:40.616790 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:05:44.116059 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:44.116027 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:44.249363 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:44.249318 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:48.710627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:48.710593 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:53.195076 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:53.195003 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:55.066582 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:55.066552 4441 scope.go:122] "RemoveContainer" containerID="8adbb6fb5861eea86c735ae68a175bed0443a600784a0787a7d8336cb8c960e8" Nov 02 22:05:57.684865 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:57.684832 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:58.389653 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:05:58.389612 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:02.193334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:02.193256 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:06.693524 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:06.693487 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:10.614498 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:06:10.614460 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:06:11.279726 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:11.279671 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:15.788746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:15.788713 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:20.315339 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:20.315306 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:24.799036 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:24.798997 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:29.379631 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:29.379597 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:33.882609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:33.882575 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:38.389449 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:38.389414 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:39.043389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:39.043358 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc"] Nov 02 22:06:39.072942 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:39.072914 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd8b1d3-mwfkc"] Nov 02 22:06:40.043480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:40.043448 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4"] Nov 02 22:06:40.090749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:40.090717 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27f2f5c-nrkq4"] Nov 02 22:06:40.583291 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:40.583260 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9b40c8-bb5d-44f7-824b-273d24bea51f" path="/var/lib/kubelet/pods/5f9b40c8-bb5d-44f7-824b-273d24bea51f/volumes" Nov 02 22:06:40.583700 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:40.583686 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cdc737-0552-4fde-942c-dd7b0f71a635" path="/var/lib/kubelet/pods/93cdc737-0552-4fde-942c-dd7b0f71a635/volumes" Nov 02 22:06:40.615500 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:06:40.615464 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:06:42.888600 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:42.888565 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:47.492403 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:47.492370 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:52.000961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:52.000927 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:55.099563 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:55.099531 4441 scope.go:122] "RemoveContainer" containerID="143aeae8f762ac2d0ce4f1e0fbe5c222bd21ccec845007b01b421c5efa83b343" Nov 02 22:06:55.111809 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:55.111791 4441 scope.go:122] "RemoveContainer" containerID="dd3e876a1ff2fa19714cac966af4e8f30bdc04fe1f2cf81f5cb2521d51636985" Nov 02 22:06:56.481103 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:56.481067 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:58.721159 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:06:58.721129 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:00.986737 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:00.986704 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:05.488171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:05.488117 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:09.982849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:09.982815 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:10.615928 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:07:10.615893 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:07:14.485750 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:14.485718 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:19.061510 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:19.061477 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:20.045465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:20.045429 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5"] Nov 02 22:07:20.056569 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:20.056541 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fac650-xn4z5"] Nov 02 22:07:20.582722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:20.582686 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d15082-8d63-409b-bc2c-fc2159d7cffa" path="/var/lib/kubelet/pods/47d15082-8d63-409b-bc2c-fc2159d7cffa/volumes" Nov 02 22:07:22.042508 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:22.042472 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m"] Nov 02 22:07:22.089423 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:22.089394 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b3f1ed1-t247m"] Nov 02 22:07:22.583410 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:22.583379 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f37c3c-f91e-43c8-a8f2-33335d721702" path="/var/lib/kubelet/pods/e5f37c3c-f91e-43c8-a8f2-33335d721702/volumes" Nov 02 22:07:23.570799 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:23.570768 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:28.080935 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:28.080903 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:32.584456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:32.584379 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:37.081101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:37.081062 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:40.614688 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:07:40.614650 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:07:41.603890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:41.603853 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:46.085898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:46.085863 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:50.010317 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:50.010284 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:50.127285 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:50.127253 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:50.580647 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:50.580601 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:51.707767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:51.707736 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:07:51.715070 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:51.715042 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:07:51.716383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:51.716364 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:07:51.726507 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:51.726487 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:07:51.730137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:51.730116 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:07:51.737427 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:51.737409 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:07:55.083392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:55.083355 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:55.144286 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:55.144252 4441 scope.go:122] "RemoveContainer" containerID="0af6bc83c61080003038ca2880e8d79660138c40637353231ad23376c25ce32c" Nov 02 22:07:55.154319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:55.154299 4441 scope.go:122] "RemoveContainer" containerID="a47b46d34d17c8818edcdaf2257080fe5855d64bbcc5349879816b904071827d" Nov 02 22:07:55.163197 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:55.163177 4441 scope.go:122] "RemoveContainer" containerID="1d2fb41f71d9ea35e272300c4cbd390923d21d04c539bd590ce10220d7969ec5" Nov 02 22:07:57.508686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:57.508626 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:57.621358 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:57.621277 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:59.047627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:07:59.047596 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:00.410762 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:00.410731 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:04.083549 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:04.083514 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:06.998829 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:06.998797 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:07.111743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:07.111709 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:08.578500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:08.578468 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:10.618250 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:08:10.618213 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:08:11.293826 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.293790 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn"] Nov 02 22:08:11.299312 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.299289 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.348791 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.348766 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zhn\" (UniqueName: \"kubernetes.io/projected/02f57b51-1eb6-4f06-8f31-f345b032d8dd-kube-api-access-88zhn\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.349004 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.348851 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.349093 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.349036 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.396231 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.396200 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:08:11.396388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.396244 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:08:11.396388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.396277 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:08:11.396388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.396337 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:08:11.399464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.399442 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn"] Nov 02 22:08:11.449850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.449823 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.450010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.449864 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zhn\" (UniqueName: \"kubernetes.io/projected/02f57b51-1eb6-4f06-8f31-f345b032d8dd-kube-api-access-88zhn\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.450010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.450001 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.450594 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.450571 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.452166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.452148 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.503247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.503221 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zhn\" (UniqueName: \"kubernetes.io/projected/02f57b51-1eb6-4f06-8f31-f345b032d8dd-kube-api-access-88zhn\") pod \"maas-api-revoke-sub-62c3724d-czgxn\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.612140 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.612116 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:11.976666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:11.976632 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn"] Nov 02 22:08:11.977403 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:08:11.977376 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f57b51_1eb6_4f06_8f31_f345b032d8dd.slice/crio-1b73e58deeab4bb0f850859f555bf6a02327bfe4228ac0347b92adc02cbefb20 WatchSource:0}: Error finding container 1b73e58deeab4bb0f850859f555bf6a02327bfe4228ac0347b92adc02cbefb20: Status 404 returned error can't find the container with id 1b73e58deeab4bb0f850859f555bf6a02327bfe4228ac0347b92adc02cbefb20 Nov 02 22:08:12.621815 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:12.621777 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" event={"ID":"02f57b51-1eb6-4f06-8f31-f345b032d8dd","Type":"ContainerStarted","Data":"69649c6d6b19c9e70e5295dd88fa569a94464a53f8878d9a6c604f051eba6ff8"} Nov 02 22:08:12.622051 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:12.621842 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" event={"ID":"02f57b51-1eb6-4f06-8f31-f345b032d8dd","Type":"ContainerStarted","Data":"1b73e58deeab4bb0f850859f555bf6a02327bfe4228ac0347b92adc02cbefb20"} Nov 02 22:08:12.730553 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:12.730499 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" podStartSLOduration=1.7304823 podStartE2EDuration="1.7304823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:12.7288475 +0000 UTC m=+3322.284475925" watchObservedRunningTime="2026-11-02 22:08:12.7304823 +0000 UTC m=+3322.286110637" Nov 02 22:08:13.086643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:13.086608 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:13.629497 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:13.629462 4441 generic.go:356] "Generic (PLEG): container finished" podID="02f57b51-1eb6-4f06-8f31-f345b032d8dd" containerID="69649c6d6b19c9e70e5295dd88fa569a94464a53f8878d9a6c604f051eba6ff8" exitCode=0 Nov 02 22:08:13.629497 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:13.629501 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" event={"ID":"02f57b51-1eb6-4f06-8f31-f345b032d8dd","Type":"ContainerDied","Data":"69649c6d6b19c9e70e5295dd88fa569a94464a53f8878d9a6c604f051eba6ff8"} Nov 02 22:08:14.801788 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.801755 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:14.885997 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.885953 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f57b51-1eb6-4f06-8f31-f345b032d8dd-kube-api-access-88zhn\" (UniqueName: \"kubernetes.io/projected/02f57b51-1eb6-4f06-8f31-f345b032d8dd-kube-api-access-88zhn\") pod \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " Nov 02 22:08:14.886167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.886054 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-xks-ca\") pod \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " Nov 02 22:08:14.886167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.886118 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-openshift-ca\") pod \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\" (UID: \"02f57b51-1eb6-4f06-8f31-f345b032d8dd\") " Nov 02 22:08:14.886711 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.886687 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-openshift-ca" pod "02f57b51-1eb6-4f06-8f31-f345b032d8dd" (UID: "02f57b51-1eb6-4f06-8f31-f345b032d8dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:14.888206 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.888146 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-xks-ca" pod "02f57b51-1eb6-4f06-8f31-f345b032d8dd" (UID: "02f57b51-1eb6-4f06-8f31-f345b032d8dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:14.888206 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.888161 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f57b51-1eb6-4f06-8f31-f345b032d8dd-kube-api-access-88zhn" pod "02f57b51-1eb6-4f06-8f31-f345b032d8dd" (UID: "02f57b51-1eb6-4f06-8f31-f345b032d8dd"). InnerVolumeSpecName "kube-api-access-88zhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:14.987605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.987563 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:14.987605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.987601 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f57b51-1eb6-4f06-8f31-f345b032d8dd-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:14.987605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:14.987614 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88zhn\" (UniqueName: \"kubernetes.io/projected/02f57b51-1eb6-4f06-8f31-f345b032d8dd-kube-api-access-88zhn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:15.645735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:15.645699 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" Nov 02 22:08:15.645735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:15.645724 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn" event={"ID":"02f57b51-1eb6-4f06-8f31-f345b032d8dd","Type":"ContainerDied","Data":"1b73e58deeab4bb0f850859f555bf6a02327bfe4228ac0347b92adc02cbefb20"} Nov 02 22:08:15.646001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:15.645771 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b73e58deeab4bb0f850859f555bf6a02327bfe4228ac0347b92adc02cbefb20" Nov 02 22:08:17.682485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:17.682449 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:22.189397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:22.189359 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:23.944680 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:23.944651 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:24.898412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.898370 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks"] Nov 02 22:08:24.898988 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.898955 4441 cpu_manager.go:414] "containerMap: removing container" podUID="02f57b51-1eb6-4f06-8f31-f345b032d8dd" containerName="revoke-keys" Nov 02 22:08:24.899045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.898991 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f57b51-1eb6-4f06-8f31-f345b032d8dd" containerName="revoke-keys" Nov 02 22:08:24.899143 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.899132 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f57b51-1eb6-4f06-8f31-f345b032d8dd" containerName="revoke-keys" Nov 02 22:08:24.902714 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.902687 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:24.953782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.953749 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks"] Nov 02 22:08:24.982533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.982499 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:24.982702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.982628 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpldl\" (UniqueName: \"kubernetes.io/projected/25f47e60-71d6-49ac-9720-4bc57343bd3e-kube-api-access-gpldl\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:24.982808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.982790 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:24.996604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.996565 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:08:24.996604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.996594 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:08:24.996604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.996570 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:08:24.996820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:24.996570 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:08:25.084146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.084111 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpldl\" (UniqueName: \"kubernetes.io/projected/25f47e60-71d6-49ac-9720-4bc57343bd3e-kube-api-access-gpldl\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:25.084357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.084213 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:25.084357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.084240 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:25.084914 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.084887 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:25.086881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.086824 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:25.137956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.137926 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpldl\" (UniqueName: \"kubernetes.io/projected/25f47e60-71d6-49ac-9720-4bc57343bd3e-kube-api-access-gpldl\") pod \"maas-api-revoke-sub-815733d1-qqcks\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:25.216914 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.216838 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:25.597194 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:08:25.597160 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f47e60_71d6_49ac_9720_4bc57343bd3e.slice/crio-d88ca62cffcefd42bc3557dffbae7c6e207794f8228151a7c516b416c06521ce WatchSource:0}: Error finding container d88ca62cffcefd42bc3557dffbae7c6e207794f8228151a7c516b416c06521ce: Status 404 returned error can't find the container with id d88ca62cffcefd42bc3557dffbae7c6e207794f8228151a7c516b416c06521ce Nov 02 22:08:25.597902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.597876 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks"] Nov 02 22:08:25.728288 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:25.728252 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" event={"ID":"25f47e60-71d6-49ac-9720-4bc57343bd3e","Type":"ContainerStarted","Data":"d88ca62cffcefd42bc3557dffbae7c6e207794f8228151a7c516b416c06521ce"} Nov 02 22:08:26.684754 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:26.684671 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:26.737259 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:26.737215 4441 generic.go:356] "Generic (PLEG): container finished" podID="25f47e60-71d6-49ac-9720-4bc57343bd3e" containerID="62706814850bfe9771481a91b5e300e3b26a5419ae92479858b36a1192ab6061" exitCode=0 Nov 02 22:08:26.742051 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:26.741989 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" event={"ID":"25f47e60-71d6-49ac-9720-4bc57343bd3e","Type":"ContainerDied","Data":"62706814850bfe9771481a91b5e300e3b26a5419ae92479858b36a1192ab6061"} Nov 02 22:08:27.886792 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:27.886764 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:28.014157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.014076 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-xks-ca\") pod \"25f47e60-71d6-49ac-9720-4bc57343bd3e\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " Nov 02 22:08:28.014298 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.014268 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-openshift-ca\") pod \"25f47e60-71d6-49ac-9720-4bc57343bd3e\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " Nov 02 22:08:28.014399 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.014382 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f47e60-71d6-49ac-9720-4bc57343bd3e-kube-api-access-gpldl\" (UniqueName: \"kubernetes.io/projected/25f47e60-71d6-49ac-9720-4bc57343bd3e-kube-api-access-gpldl\") pod \"25f47e60-71d6-49ac-9720-4bc57343bd3e\" (UID: \"25f47e60-71d6-49ac-9720-4bc57343bd3e\") " Nov 02 22:08:28.014672 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.014632 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-openshift-ca" pod "25f47e60-71d6-49ac-9720-4bc57343bd3e" (UID: "25f47e60-71d6-49ac-9720-4bc57343bd3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:28.014804 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.014735 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:28.016323 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.016289 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-xks-ca" pod "25f47e60-71d6-49ac-9720-4bc57343bd3e" (UID: "25f47e60-71d6-49ac-9720-4bc57343bd3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:28.016483 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.016385 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f47e60-71d6-49ac-9720-4bc57343bd3e-kube-api-access-gpldl" pod "25f47e60-71d6-49ac-9720-4bc57343bd3e" (UID: "25f47e60-71d6-49ac-9720-4bc57343bd3e"). InnerVolumeSpecName "kube-api-access-gpldl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:28.115936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.115898 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpldl\" (UniqueName: \"kubernetes.io/projected/25f47e60-71d6-49ac-9720-4bc57343bd3e-kube-api-access-gpldl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:28.116124 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.115941 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f47e60-71d6-49ac-9720-4bc57343bd3e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:28.760080 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.760047 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" Nov 02 22:08:28.760258 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.760080 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks" event={"ID":"25f47e60-71d6-49ac-9720-4bc57343bd3e","Type":"ContainerDied","Data":"d88ca62cffcefd42bc3557dffbae7c6e207794f8228151a7c516b416c06521ce"} Nov 02 22:08:28.760258 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:28.760136 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d88ca62cffcefd42bc3557dffbae7c6e207794f8228151a7c516b416c06521ce" Nov 02 22:08:30.578921 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:30.578876 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:31.189115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:31.189085 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:35.743115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:35.743082 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:39.045959 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:39.045926 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr"] Nov 02 22:08:39.093394 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:39.093339 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x"] Nov 02 22:08:39.096255 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:39.096224 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06073ecc-jgwsr"] Nov 02 22:08:39.098803 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:39.098779 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc185c-6zt4x"] Nov 02 22:08:40.193212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:40.193181 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:40.582844 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:40.582810 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e26b62-b3da-40d2-b692-6241fd2f0b62" path="/var/lib/kubelet/pods/72e26b62-b3da-40d2-b692-6241fd2f0b62/volumes" Nov 02 22:08:40.583256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:40.583241 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f4cf9c-ab75-4f17-a82a-f125cfc4489b" path="/var/lib/kubelet/pods/a4f4cf9c-ab75-4f17-a82a-f125cfc4489b/volumes" Nov 02 22:08:40.614441 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:08:40.614405 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:08:44.685542 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:44.685511 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:49.184688 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:49.184656 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:50.067070 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.067034 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk"] Nov 02 22:08:50.075210 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.075175 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:50.610203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.610165 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg"] Nov 02 22:08:50.610740 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.610720 4441 cpu_manager.go:414] "containerMap: removing container" podUID="25f47e60-71d6-49ac-9720-4bc57343bd3e" containerName="revoke-keys" Nov 02 22:08:50.610838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.610743 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f47e60-71d6-49ac-9720-4bc57343bd3e" containerName="revoke-keys" Nov 02 22:08:50.610902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.610889 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f47e60-71d6-49ac-9720-4bc57343bd3e" containerName="revoke-keys" Nov 02 22:08:50.615463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.615441 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.665005 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.664948 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:08:50.665175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.665072 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:08:50.665175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.665147 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:08:50.666197 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.666172 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg"] Nov 02 22:08:50.709460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.709437 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:08:50.730651 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.730621 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.730764 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.730658 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jjj\" (UniqueName: \"kubernetes.io/projected/0e47ca04-2334-4be4-a60d-18e3bdad40d1-kube-api-access-h5jjj\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.730822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.730791 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.831950 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.831928 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.832073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.832042 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.832073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.832064 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jjj\" (UniqueName: \"kubernetes.io/projected/0e47ca04-2334-4be4-a60d-18e3bdad40d1-kube-api-access-h5jjj\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.832677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.832656 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.834327 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.834301 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.885850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.885789 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jjj\" (UniqueName: \"kubernetes.io/projected/0e47ca04-2334-4be4-a60d-18e3bdad40d1-kube-api-access-h5jjj\") pod \"maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:50.929129 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:50.929104 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:51.295492 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:51.295462 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg"] Nov 02 22:08:51.296434 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:08:51.296404 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e47ca04_2334_4be4_a60d_18e3bdad40d1.slice/crio-e646c618ded8e7fa5b5d9116a3e9b961583e1715514f8e48f2e5f4fd1a152308 WatchSource:0}: Error finding container e646c618ded8e7fa5b5d9116a3e9b961583e1715514f8e48f2e5f4fd1a152308: Status 404 returned error can't find the container with id e646c618ded8e7fa5b5d9116a3e9b961583e1715514f8e48f2e5f4fd1a152308 Nov 02 22:08:51.919712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:51.919622 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" event={"ID":"0e47ca04-2334-4be4-a60d-18e3bdad40d1","Type":"ContainerStarted","Data":"0f7bbb16984989c1a3661deee3a365db20447bed9c43a7b7c7dd7dff15bc59fc"} Nov 02 22:08:51.919712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:51.919671 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" event={"ID":"0e47ca04-2334-4be4-a60d-18e3bdad40d1","Type":"ContainerStarted","Data":"e646c618ded8e7fa5b5d9116a3e9b961583e1715514f8e48f2e5f4fd1a152308"} Nov 02 22:08:52.028089 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:52.028025 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" podStartSLOduration=2.028011865 podStartE2EDuration="2.028011865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:52.027316917 +0000 UTC m=+3361.582945255" watchObservedRunningTime="2026-11-02 22:08:52.028011865 +0000 UTC m=+3361.583640202" Nov 02 22:08:52.927894 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:52.927860 4441 generic.go:356] "Generic (PLEG): container finished" podID="0e47ca04-2334-4be4-a60d-18e3bdad40d1" containerID="0f7bbb16984989c1a3661deee3a365db20447bed9c43a7b7c7dd7dff15bc59fc" exitCode=0 Nov 02 22:08:52.928317 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:52.927948 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" event={"ID":"0e47ca04-2334-4be4-a60d-18e3bdad40d1","Type":"ContainerDied","Data":"0f7bbb16984989c1a3661deee3a365db20447bed9c43a7b7c7dd7dff15bc59fc"} Nov 02 22:08:54.087220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.087194 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:54.190734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.190652 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:54.260519 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.260489 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-openshift-ca\") pod \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " Nov 02 22:08:54.260678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.260587 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e47ca04-2334-4be4-a60d-18e3bdad40d1-kube-api-access-h5jjj\" (UniqueName: \"kubernetes.io/projected/0e47ca04-2334-4be4-a60d-18e3bdad40d1-kube-api-access-h5jjj\") pod \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " Nov 02 22:08:54.260678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.260628 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-xks-ca\") pod \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\" (UID: \"0e47ca04-2334-4be4-a60d-18e3bdad40d1\") " Nov 02 22:08:54.260904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.260880 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-openshift-ca" pod "0e47ca04-2334-4be4-a60d-18e3bdad40d1" (UID: "0e47ca04-2334-4be4-a60d-18e3bdad40d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:54.261113 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.261088 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:54.262705 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.262677 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-xks-ca" pod "0e47ca04-2334-4be4-a60d-18e3bdad40d1" (UID: "0e47ca04-2334-4be4-a60d-18e3bdad40d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:54.262819 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.262681 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e47ca04-2334-4be4-a60d-18e3bdad40d1-kube-api-access-h5jjj" pod "0e47ca04-2334-4be4-a60d-18e3bdad40d1" (UID: "0e47ca04-2334-4be4-a60d-18e3bdad40d1"). InnerVolumeSpecName "kube-api-access-h5jjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:54.361908 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.361877 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jjj\" (UniqueName: \"kubernetes.io/projected/0e47ca04-2334-4be4-a60d-18e3bdad40d1-kube-api-access-h5jjj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:54.361908 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.361904 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e47ca04-2334-4be4-a60d-18e3bdad40d1-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:54.942799 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.942719 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" event={"ID":"0e47ca04-2334-4be4-a60d-18e3bdad40d1","Type":"ContainerDied","Data":"e646c618ded8e7fa5b5d9116a3e9b961583e1715514f8e48f2e5f4fd1a152308"} Nov 02 22:08:54.942799 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.942743 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg" Nov 02 22:08:54.942799 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:54.942774 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e646c618ded8e7fa5b5d9116a3e9b961583e1715514f8e48f2e5f4fd1a152308" Nov 02 22:08:55.196937 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:55.196854 4441 scope.go:122] "RemoveContainer" containerID="7e1cad03fe0e7449226abb1a499a3cad43da78ebb8d5a7099f59c46c3a2ef9d1" Nov 02 22:08:55.206921 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:55.206892 4441 scope.go:122] "RemoveContainer" containerID="3f821c66aa5a356fde7ff16d986f91eff4e42d63e5b432fea8ef3c2c9ce38e07" Nov 02 22:08:55.215458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:55.215440 4441 scope.go:122] "RemoveContainer" containerID="d702d0a30f102fba7056042b2386f3bd8b0ec9d23228428fd71b2938cf89f5f8" Nov 02 22:08:55.224606 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:55.224577 4441 scope.go:122] "RemoveContainer" containerID="e48a4310e1fc1b7253c4efcc5ca2da2ffc865b78a45faf3120558e84502eed7e" Nov 02 22:08:55.232826 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:55.232808 4441 scope.go:122] "RemoveContainer" containerID="7759f2e24ba9d2a33404ddf1f094157f77b89ce62279a4ed1fa6f350f4f25719" Nov 02 22:08:55.240794 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:55.240771 4441 scope.go:122] "RemoveContainer" containerID="41b169cf645b2ea39428a4a7ad9e2cef8db6eae0b01300c0ceb6df49fd8b97c7" Nov 02 22:08:55.248666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:55.248647 4441 scope.go:122] "RemoveContainer" containerID="4c630bdf0e83e29976a494c9eb6aacd9a7d87931eba65e3c1ca6b14b8f6b30cb" Nov 02 22:08:58.876654 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:58.876617 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:59.346136 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:08:59.346108 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:02.095331 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:02.095239 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t"] Nov 02 22:09:02.195120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:02.195085 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g"] Nov 02 22:09:02.431123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:02.431038 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:02.549690 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:02.549656 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:03.391168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:03.391134 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:05.086596 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.086542 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" podUID="cbef99f8-743b-437c-9866-8bab0cf3c788" containerName="main" containerID="cri-o://da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916" gracePeriod=45 Nov 02 22:09:05.337197 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.337140 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:09:05.375302 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375276 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbef99f8-743b-437c-9866-8bab0cf3c788-kube-api-access-2wjjw\" (UniqueName: \"kubernetes.io/projected/cbef99f8-743b-437c-9866-8bab0cf3c788-kube-api-access-2wjjw\") pod \"cbef99f8-743b-437c-9866-8bab0cf3c788\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " Nov 02 22:09:05.375421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375403 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-home\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-home\") pod \"cbef99f8-743b-437c-9866-8bab0cf3c788\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " Nov 02 22:09:05.375464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375434 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-model-cache\") pod \"cbef99f8-743b-437c-9866-8bab0cf3c788\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " Nov 02 22:09:05.375525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375483 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbef99f8-743b-437c-9866-8bab0cf3c788-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbef99f8-743b-437c-9866-8bab0cf3c788-tls-certs\") pod \"cbef99f8-743b-437c-9866-8bab0cf3c788\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " Nov 02 22:09:05.375525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375517 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-tmp-dir\") pod \"cbef99f8-743b-437c-9866-8bab0cf3c788\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " Nov 02 22:09:05.375640 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375579 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-dshm\") pod \"cbef99f8-743b-437c-9866-8bab0cf3c788\" (UID: \"cbef99f8-743b-437c-9866-8bab0cf3c788\") " Nov 02 22:09:05.375702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375681 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-model-cache" pod "cbef99f8-743b-437c-9866-8bab0cf3c788" (UID: "cbef99f8-743b-437c-9866-8bab0cf3c788"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:05.375765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375692 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-home" pod "cbef99f8-743b-437c-9866-8bab0cf3c788" (UID: "cbef99f8-743b-437c-9866-8bab0cf3c788"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:05.375917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.375889 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-tmp-dir" pod "cbef99f8-743b-437c-9866-8bab0cf3c788" (UID: "cbef99f8-743b-437c-9866-8bab0cf3c788"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:05.376029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.376012 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:05.376087 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.376038 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:05.377490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.377466 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-dshm" pod "cbef99f8-743b-437c-9866-8bab0cf3c788" (UID: "cbef99f8-743b-437c-9866-8bab0cf3c788"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:05.377578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.377510 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbef99f8-743b-437c-9866-8bab0cf3c788-kube-api-access-2wjjw" pod "cbef99f8-743b-437c-9866-8bab0cf3c788" (UID: "cbef99f8-743b-437c-9866-8bab0cf3c788"). InnerVolumeSpecName "kube-api-access-2wjjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:05.377578 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.377531 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbef99f8-743b-437c-9866-8bab0cf3c788-tls-certs" pod "cbef99f8-743b-437c-9866-8bab0cf3c788" (UID: "cbef99f8-743b-437c-9866-8bab0cf3c788"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:05.476999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.476954 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjjw\" (UniqueName: \"kubernetes.io/projected/cbef99f8-743b-437c-9866-8bab0cf3c788-kube-api-access-2wjjw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:05.476999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.476991 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbef99f8-743b-437c-9866-8bab0cf3c788-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:05.476999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.477001 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:05.477213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:05.477010 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbef99f8-743b-437c-9866-8bab0cf3c788-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:06.018172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.018133 4441 generic.go:356] "Generic (PLEG): container finished" podID="cbef99f8-743b-437c-9866-8bab0cf3c788" containerID="da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916" exitCode=0 Nov 02 22:09:06.018345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.018180 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" Nov 02 22:09:06.018345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.018210 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" event={"ID":"cbef99f8-743b-437c-9866-8bab0cf3c788","Type":"ContainerDied","Data":"da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916"} Nov 02 22:09:06.018345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.018254 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" event={"ID":"cbef99f8-743b-437c-9866-8bab0cf3c788","Type":"ContainerDied","Data":"ae0d8270827f1eb374fe9bee7b24b4931e88a80ac8e03b1c196005c5c3ee2fcb"} Nov 02 22:09:06.018345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.018272 4441 scope.go:122] "RemoveContainer" containerID="da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916" Nov 02 22:09:06.028799 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.028780 4441 scope.go:122] "RemoveContainer" containerID="da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916" Nov 02 22:09:06.029076 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:06.029058 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916\": container with ID starting with da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916 not found: ID does not exist" containerID="da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916" Nov 02 22:09:06.029134 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.029085 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916"} err="failed to get container status \"da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916\": rpc error: code = NotFound desc = could not find container \"da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916\": container with ID starting with da0361a36ea927d710abf8660f1b98b62906408d371fca2c8108c0402c861916 not found: ID does not exist" Nov 02 22:09:06.173012 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.172960 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk"] Nov 02 22:09:06.176602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.176569 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk"] Nov 02 22:09:06.317102 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.317057 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-49c06d/test-model-49c06d-kserve-58f654bfd7-tfxnk" podUID="cbef99f8-743b-437c-9866-8bab0cf3c788" containerName="main" probeResult="failure" output="Get \"https://10.133.1.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:09:06.583448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:06.583374 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbef99f8-743b-437c-9866-8bab0cf3c788" path="/var/lib/kubelet/pods/cbef99f8-743b-437c-9866-8bab0cf3c788/volumes" Nov 02 22:09:07.095585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:07.095546 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" podUID="2f516506-1916-4436-bc71-7da08d430134" containerName="payload-processing" containerID="cri-o://92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62" gracePeriod=25 Nov 02 22:09:07.195491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:07.195453 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" containerName="payload-pre-processing" containerID="cri-o://57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7" gracePeriod=25 Nov 02 22:09:07.384145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:07.384054 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:07.384145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:07.384086 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" podUID="2f516506-1916-4436-bc71-7da08d430134" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:07.903905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:07.903872 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:10.618002 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:10.617942 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:09:11.834666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:11.834631 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:13.284572 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:13.284530 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:13.497458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:13.497412 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:13.620936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:13.620905 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:13.866296 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:13.866258 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg"] Nov 02 22:09:13.913624 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:13.913543 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4a6b9-spvsg"] Nov 02 22:09:14.262590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:14.262502 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:14.583119 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:14.583090 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e47ca04-2334-4be4-a60d-18e3bdad40d1" path="/var/lib/kubelet/pods/0e47ca04-2334-4be4-a60d-18e3bdad40d1/volumes" Nov 02 22:09:17.384277 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:17.384237 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:17.384819 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:17.384247 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" podUID="2f516506-1916-4436-bc71-7da08d430134" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:18.293920 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:18.293881 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:19.975704 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:19.975671 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:20.101184 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:20.101154 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:20.302772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:20.302694 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7"] Nov 02 22:09:20.302940 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:20.302921 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" containerID="cri-o://9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68" gracePeriod=30 Nov 02 22:09:20.325729 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:20.325703 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:21.157015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.156975 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9"] Nov 02 22:09:21.157579 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.157557 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0e47ca04-2334-4be4-a60d-18e3bdad40d1" containerName="revoke-keys" Nov 02 22:09:21.157678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.157581 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e47ca04-2334-4be4-a60d-18e3bdad40d1" containerName="revoke-keys" Nov 02 22:09:21.157678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.157609 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cbef99f8-743b-437c-9866-8bab0cf3c788" containerName="main" Nov 02 22:09:21.157678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.157617 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbef99f8-743b-437c-9866-8bab0cf3c788" containerName="main" Nov 02 22:09:21.157843 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.157744 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e47ca04-2334-4be4-a60d-18e3bdad40d1" containerName="revoke-keys" Nov 02 22:09:21.157843 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.157759 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbef99f8-743b-437c-9866-8bab0cf3c788" containerName="main" Nov 02 22:09:21.162597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.162575 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.213332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.213308 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9"] Nov 02 22:09:21.226371 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.226342 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.226474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.226397 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.226517 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.226468 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxpm5\" (UniqueName: \"kubernetes.io/projected/6f51f46e-d646-446b-bf06-b38a5c4b53ef-kube-api-access-hxpm5\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.256383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.256356 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:09:21.256485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.256389 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:09:21.256485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.256414 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:09:21.258819 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.258803 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:09:21.327060 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.327039 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.327175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.327072 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.327175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.327094 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxpm5\" (UniqueName: \"kubernetes.io/projected/6f51f46e-d646-446b-bf06-b38a5c4b53ef-kube-api-access-hxpm5\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.327656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.327637 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.329409 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.329390 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.380527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.380499 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxpm5\" (UniqueName: \"kubernetes.io/projected/6f51f46e-d646-446b-bf06-b38a5c4b53ef-kube-api-access-hxpm5\") pod \"maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.475270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.475218 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:21.836981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:21.836943 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9"] Nov 02 22:09:21.837734 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:09:21.837707 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f51f46e_d646_446b_bf06_b38a5c4b53ef.slice/crio-21137b89d8d590676a33900f8dd3ebc57f143b6ccc758248eccfdddd6c462c5b WatchSource:0}: Error finding container 21137b89d8d590676a33900f8dd3ebc57f143b6ccc758248eccfdddd6c462c5b: Status 404 returned error can't find the container with id 21137b89d8d590676a33900f8dd3ebc57f143b6ccc758248eccfdddd6c462c5b Nov 02 22:09:22.122015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:22.121944 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" event={"ID":"6f51f46e-d646-446b-bf06-b38a5c4b53ef","Type":"ContainerStarted","Data":"21137b89d8d590676a33900f8dd3ebc57f143b6ccc758248eccfdddd6c462c5b"} Nov 02 22:09:22.925309 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:22.925271 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:23.132664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:23.132633 4441 generic.go:356] "Generic (PLEG): container finished" podID="6f51f46e-d646-446b-bf06-b38a5c4b53ef" containerID="a8998f84173c0da551e74d8cb52d54f27095dacb647fea7a972e4afa65db2617" exitCode=0 Nov 02 22:09:23.132842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:23.132716 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" event={"ID":"6f51f46e-d646-446b-bf06-b38a5c4b53ef","Type":"ContainerDied","Data":"a8998f84173c0da551e74d8cb52d54f27095dacb647fea7a972e4afa65db2617"} Nov 02 22:09:24.279886 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.279859 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:24.359792 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.359761 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-openshift-ca\") pod \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " Nov 02 22:09:24.359935 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.359812 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-xks-ca\") pod \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " Nov 02 22:09:24.359935 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.359913 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f51f46e-d646-446b-bf06-b38a5c4b53ef-kube-api-access-hxpm5\" (UniqueName: \"kubernetes.io/projected/6f51f46e-d646-446b-bf06-b38a5c4b53ef-kube-api-access-hxpm5\") pod \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\" (UID: \"6f51f46e-d646-446b-bf06-b38a5c4b53ef\") " Nov 02 22:09:24.360177 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.360156 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-openshift-ca" pod "6f51f46e-d646-446b-bf06-b38a5c4b53ef" (UID: "6f51f46e-d646-446b-bf06-b38a5c4b53ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:24.360348 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.360324 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:24.362060 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.362032 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f51f46e-d646-446b-bf06-b38a5c4b53ef-kube-api-access-hxpm5" pod "6f51f46e-d646-446b-bf06-b38a5c4b53ef" (UID: "6f51f46e-d646-446b-bf06-b38a5c4b53ef"). InnerVolumeSpecName "kube-api-access-hxpm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:24.362157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.362131 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-xks-ca" pod "6f51f46e-d646-446b-bf06-b38a5c4b53ef" (UID: "6f51f46e-d646-446b-bf06-b38a5c4b53ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:24.461177 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.461101 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f51f46e-d646-446b-bf06-b38a5c4b53ef-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:24.461177 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:24.461129 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxpm5\" (UniqueName: \"kubernetes.io/projected/6f51f46e-d646-446b-bf06-b38a5c4b53ef-kube-api-access-hxpm5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.147561 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.147528 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" Nov 02 22:09:25.147561 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.147539 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9" event={"ID":"6f51f46e-d646-446b-bf06-b38a5c4b53ef","Type":"ContainerDied","Data":"21137b89d8d590676a33900f8dd3ebc57f143b6ccc758248eccfdddd6c462c5b"} Nov 02 22:09:25.147777 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.147582 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21137b89d8d590676a33900f8dd3ebc57f143b6ccc758248eccfdddd6c462c5b" Nov 02 22:09:25.565093 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.565071 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 22:09:25.671946 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.671860 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-token\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-token\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.671946 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.671908 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-credential-socket\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.671947 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-kube-api-access-lzbsx\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-kube-api-access-lzbsx\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672053 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-podinfo\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672133 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-socket\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672191 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-data\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672457 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672283 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-credential-socket" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:25.672457 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672319 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-envoy\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672457 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672404 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-certs\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672457 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672435 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istiod-ca-cert\") pod \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\" (UID: \"2aac1cb0-9792-4d93-b66e-259a1d6a6500\") " Nov 02 22:09:25.672697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672502 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-socket" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:25.672697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672548 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-data" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:25.672812 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672701 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-certs" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:25.672911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.672876 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istiod-ca-cert" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:25.673121 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.673096 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.673213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.673125 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.673213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.673157 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.673213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.673170 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.673213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.673185 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.674753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.674728 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-kube-api-access-lzbsx" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "kube-api-access-lzbsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:25.674938 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.674917 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-envoy" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:25.675066 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.675047 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-podinfo" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:09:25.675141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.675102 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-token" pod "2aac1cb0-9792-4d93-b66e-259a1d6a6500" (UID: "2aac1cb0-9792-4d93-b66e-259a1d6a6500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:25.773885 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.773858 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.773885 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.773883 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbsx\" (UniqueName: \"kubernetes.io/projected/2aac1cb0-9792-4d93-b66e-259a1d6a6500-kube-api-access-lzbsx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.774101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.773892 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:25.774101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:25.773901 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aac1cb0-9792-4d93-b66e-259a1d6a6500-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:26.155493 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.155456 4441 generic.go:356] "Generic (PLEG): container finished" podID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerID="9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68" exitCode=0 Nov 02 22:09:26.155697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.155511 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" Nov 02 22:09:26.155697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.155532 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" event={"ID":"2aac1cb0-9792-4d93-b66e-259a1d6a6500","Type":"ContainerDied","Data":"9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68"} Nov 02 22:09:26.155697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.155588 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7" event={"ID":"2aac1cb0-9792-4d93-b66e-259a1d6a6500","Type":"ContainerDied","Data":"0ae2c4ed015d90fdcf6d929c9a4f24b52f1849cd0ef95ce2fb898a40516b4761"} Nov 02 22:09:26.155697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.155610 4441 scope.go:122] "RemoveContainer" containerID="9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68" Nov 02 22:09:26.166162 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.166145 4441 scope.go:122] "RemoveContainer" containerID="f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc" Nov 02 22:09:26.174943 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.174922 4441 scope.go:122] "RemoveContainer" containerID="9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68" Nov 02 22:09:26.175225 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:26.175204 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68\": container with ID starting with 9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68 not found: ID does not exist" containerID="9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68" Nov 02 22:09:26.175319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.175236 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68"} err="failed to get container status \"9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68\": rpc error: code = NotFound desc = could not find container \"9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68\": container with ID starting with 9e4b2b43193a3d15a9e6c4450a08f281105efd2db0c682ca83f0ea8f2e605a68 not found: ID does not exist" Nov 02 22:09:26.175319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.175259 4441 scope.go:122] "RemoveContainer" containerID="f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc" Nov 02 22:09:26.175540 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:26.175522 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc\": container with ID starting with f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc not found: ID does not exist" containerID="f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc" Nov 02 22:09:26.175581 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.175546 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc"} err="failed to get container status \"f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc\": rpc error: code = NotFound desc = could not find container \"f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc\": container with ID starting with f63e7ce46a9635d12d86776e5a0abde8d696876fc59ab0cd86e65ef8a73cb7dc not found: ID does not exist" Nov 02 22:09:26.311789 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.311756 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7"] Nov 02 22:09:26.359189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.359158 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c4a6b9-openshift-default-65db8fb597-2lkb7"] Nov 02 22:09:26.582837 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:26.582798 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" path="/var/lib/kubelet/pods/2aac1cb0-9792-4d93-b66e-259a1d6a6500/volumes" Nov 02 22:09:27.351759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.351735 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:09:27.438331 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.438311 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:09:27.491352 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.491326 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f516506-1916-4436-bc71-7da08d430134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f516506-1916-4436-bc71-7da08d430134-plugins-config-volume\") pod \"2f516506-1916-4436-bc71-7da08d430134\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " Nov 02 22:09:27.491504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.491427 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f516506-1916-4436-bc71-7da08d430134-kube-api-access-g2lrd\" (UniqueName: \"kubernetes.io/projected/2f516506-1916-4436-bc71-7da08d430134-kube-api-access-g2lrd\") pod \"2f516506-1916-4436-bc71-7da08d430134\" (UID: \"2f516506-1916-4436-bc71-7da08d430134\") " Nov 02 22:09:27.491786 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.491756 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f516506-1916-4436-bc71-7da08d430134-plugins-config-volume" pod "2f516506-1916-4436-bc71-7da08d430134" (UID: "2f516506-1916-4436-bc71-7da08d430134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:27.493499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.493475 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f516506-1916-4436-bc71-7da08d430134-kube-api-access-g2lrd" pod "2f516506-1916-4436-bc71-7da08d430134" (UID: "2f516506-1916-4436-bc71-7da08d430134"). InnerVolumeSpecName "kube-api-access-g2lrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:27.592753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.592724 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29544df9-e93d-43fd-93fc-dacbc538008a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29544df9-e93d-43fd-93fc-dacbc538008a-plugins-config-volume\") pod \"29544df9-e93d-43fd-93fc-dacbc538008a\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " Nov 02 22:09:27.593229 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.592824 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29544df9-e93d-43fd-93fc-dacbc538008a-kube-api-access-hq7xj\" (UniqueName: \"kubernetes.io/projected/29544df9-e93d-43fd-93fc-dacbc538008a-kube-api-access-hq7xj\") pod \"29544df9-e93d-43fd-93fc-dacbc538008a\" (UID: \"29544df9-e93d-43fd-93fc-dacbc538008a\") " Nov 02 22:09:27.593229 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.593064 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f516506-1916-4436-bc71-7da08d430134-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:27.593229 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.593076 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2lrd\" (UniqueName: \"kubernetes.io/projected/2f516506-1916-4436-bc71-7da08d430134-kube-api-access-g2lrd\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:27.593229 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.593082 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29544df9-e93d-43fd-93fc-dacbc538008a-plugins-config-volume" pod "29544df9-e93d-43fd-93fc-dacbc538008a" (UID: "29544df9-e93d-43fd-93fc-dacbc538008a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:27.594898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.594875 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29544df9-e93d-43fd-93fc-dacbc538008a-kube-api-access-hq7xj" pod "29544df9-e93d-43fd-93fc-dacbc538008a" (UID: "29544df9-e93d-43fd-93fc-dacbc538008a"). InnerVolumeSpecName "kube-api-access-hq7xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:27.685506 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.685416 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:27.693607 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.693572 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29544df9-e93d-43fd-93fc-dacbc538008a-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:27.693607 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:27.693598 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq7xj\" (UniqueName: \"kubernetes.io/projected/29544df9-e93d-43fd-93fc-dacbc538008a-kube-api-access-hq7xj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:28.170619 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.170588 4441 generic.go:356] "Generic (PLEG): container finished" podID="2f516506-1916-4436-bc71-7da08d430134" containerID="92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62" exitCode=0 Nov 02 22:09:28.170811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.170638 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" Nov 02 22:09:28.170811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.170679 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" event={"ID":"2f516506-1916-4436-bc71-7da08d430134","Type":"ContainerDied","Data":"92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62"} Nov 02 22:09:28.170811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.170732 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t" event={"ID":"2f516506-1916-4436-bc71-7da08d430134","Type":"ContainerDied","Data":"4f8437c4402d15e4c269e31e37fe584d8b1a6b904b3964734c663b8c40aa6a33"} Nov 02 22:09:28.170811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.170757 4441 scope.go:122] "RemoveContainer" containerID="92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62" Nov 02 22:09:28.172114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.172092 4441 generic.go:356] "Generic (PLEG): container finished" podID="29544df9-e93d-43fd-93fc-dacbc538008a" containerID="57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7" exitCode=0 Nov 02 22:09:28.172222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.172140 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:09:28.172222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.172164 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" event={"ID":"29544df9-e93d-43fd-93fc-dacbc538008a","Type":"ContainerDied","Data":"57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7"} Nov 02 22:09:28.172222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.172215 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" event={"ID":"29544df9-e93d-43fd-93fc-dacbc538008a","Type":"ContainerDied","Data":"c8aa55bbd208303a4b6719270837b30de45cfc65ecb1d7c31dead2f34dbb7b5f"} Nov 02 22:09:28.182346 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.182325 4441 scope.go:122] "RemoveContainer" containerID="92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62" Nov 02 22:09:28.182621 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:28.182600 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62\": container with ID starting with 92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62 not found: ID does not exist" containerID="92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62" Nov 02 22:09:28.182710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.182629 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62"} err="failed to get container status \"92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62\": rpc error: code = NotFound desc = could not find container \"92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62\": container with ID starting with 92767edbeca8d9625e95321ed76e99e6e570bff717912b4fe1fb0353e72c9b62 not found: ID does not exist" Nov 02 22:09:28.182710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.182646 4441 scope.go:122] "RemoveContainer" containerID="57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7" Nov 02 22:09:28.191325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.191304 4441 scope.go:122] "RemoveContainer" containerID="57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7" Nov 02 22:09:28.191568 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:28.191547 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7\": container with ID starting with 57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7 not found: ID does not exist" containerID="57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7" Nov 02 22:09:28.191612 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.191575 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7"} err="failed to get container status \"57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7\": rpc error: code = NotFound desc = could not find container \"57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7\": container with ID starting with 57fb35d23d4f1429615a37099480943762b1fe09cf3ba1c9923880e89bd01fd7 not found: ID does not exist" Nov 02 22:09:28.328548 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.328510 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t"] Nov 02 22:09:28.375229 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.375161 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4a6b9-68dd84bb55-8gv8t"] Nov 02 22:09:28.384320 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.384286 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.3:50052\" within 1s: context deadline exceeded" Nov 02 22:09:28.384426 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.384369 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g" Nov 02 22:09:28.384426 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:09:28.384403 4441 logging.go:55] [core] [Channel #9724 SubChannel #9725]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.3:50052", ServerName: "10.133.1.3:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.3:50052: operation was canceled" Nov 02 22:09:28.477538 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.477506 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g"] Nov 02 22:09:28.480952 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.480930 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4a6b9-fcb9b94f9-s642g"] Nov 02 22:09:28.583738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.583702 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" path="/var/lib/kubelet/pods/29544df9-e93d-43fd-93fc-dacbc538008a/volumes" Nov 02 22:09:28.584165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:28.584149 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f516506-1916-4436-bc71-7da08d430134" path="/var/lib/kubelet/pods/2f516506-1916-4436-bc71-7da08d430134/volumes" Nov 02 22:09:32.191375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:32.191341 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:32.559155 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:32.559127 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v"] Nov 02 22:09:32.658954 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:32.658922 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf"] Nov 02 22:09:32.892167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:32.892063 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:33.010171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:33.010116 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:36.689182 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:36.689123 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:37.559559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:37.559496 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" podUID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" containerName="payload-processing" containerID="cri-o://a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad" gracePeriod=25 Nov 02 22:09:37.660145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:37.660107 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" podUID="150790a2-98b0-44ba-ab92-06ab0a0311dd" containerName="payload-pre-processing" containerID="cri-o://74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4" gracePeriod=25 Nov 02 22:09:39.045319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:39.045281 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp"] Nov 02 22:09:39.092629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:39.092600 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz"] Nov 02 22:09:39.095218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:39.095198 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce81ab27-vbhlp"] Nov 02 22:09:39.097392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:39.097370 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4babea-l6nbz"] Nov 02 22:09:40.583597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:40.583556 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041" path="/var/lib/kubelet/pods/0a5c3cc8-edf4-4ec4-9bc1-1271d5afa041/volumes" Nov 02 22:09:40.584099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:40.584066 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90a86e1-903e-405b-a30f-13c0d73e68b3" path="/var/lib/kubelet/pods/f90a86e1-903e-405b-a30f-13c0d73e68b3/volumes" Nov 02 22:09:40.619381 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:40.619341 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:09:40.655198 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:40.655170 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:42.079455 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:42.079411 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:42.294439 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:42.294405 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:42.422959 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:42.422873 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:42.552527 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:42.552495 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9"] Nov 02 22:09:42.599400 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:42.599363 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49c06d-j95s9"] Nov 02 22:09:42.993137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:42.993097 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:43.923363 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:43.923329 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" podUID="150790a2-98b0-44ba-ab92-06ab0a0311dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:43.923755 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:43.923367 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" podUID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:44.043143 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:44.043110 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw"] Nov 02 22:09:44.074718 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:44.074688 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t"] Nov 02 22:09:44.090586 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:44.090559 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4cff178-wzvqw"] Nov 02 22:09:44.092922 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:44.092898 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23eddcca-5ph5t"] Nov 02 22:09:44.583397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:44.583363 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3782096d-ec58-4a3c-9086-9ee05e10161e" path="/var/lib/kubelet/pods/3782096d-ec58-4a3c-9086-9ee05e10161e/volumes" Nov 02 22:09:44.583773 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:44.583760 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6ddbb5-059a-4949-9d2e-5d08c29c8d26" path="/var/lib/kubelet/pods/4e6ddbb5-059a-4949-9d2e-5d08c29c8d26/volumes" Nov 02 22:09:44.584149 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:44.584137 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f51f46e-d646-446b-bf06-b38a5c4b53ef" path="/var/lib/kubelet/pods/6f51f46e-d646-446b-bf06-b38a5c4b53ef/volumes" Nov 02 22:09:46.744075 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:46.744046 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:46.914639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:46.914609 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:47.180605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:47.180562 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:48.901140 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:48.901101 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:49.034556 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:49.034523 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:49.248830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:49.248749 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv"] Nov 02 22:09:49.248999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:49.248961 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" containerID="cri-o://57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42" gracePeriod=30 Nov 02 22:09:49.257945 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:49.257916 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:49.382137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:49.382106 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:49.502616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:49.502540 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:49.723630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:49.723577 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:52.015546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:52.015508 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:53.923301 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:53.923264 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" podUID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:53.923301 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:53.923291 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" podUID="150790a2-98b0-44ba-ab92-06ab0a0311dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:09:54.503361 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.503336 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:09:54.521245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521189 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-data\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521234 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801b2255-648f-4abe-91f6-d29770490815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b2255-648f-4abe-91f6-d29770490815-istiod-ca-cert\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521267 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-socket\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521289 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-envoy\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521334 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/801b2255-648f-4abe-91f6-d29770490815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b2255-648f-4abe-91f6-d29770490815-istio-podinfo\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521361 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-credential-socket\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521371 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-data" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:54.521419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521383 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-kube-api-access-gdtsc\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-kube-api-access-gdtsc\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521546 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-istio-token\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-istio-token\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521553 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-socket" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:54.521748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521600 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-certs\") pod \"801b2255-648f-4abe-91f6-d29770490815\" (UID: \"801b2255-648f-4abe-91f6-d29770490815\") " Nov 02 22:09:54.521748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521645 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801b2255-648f-4abe-91f6-d29770490815-istiod-ca-cert" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:54.522046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521813 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-certs" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:54.522046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.521992 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.522046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.522014 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.522046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.522029 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b2255-648f-4abe-91f6-d29770490815-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.522046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.522044 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.522331 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.522134 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-credential-socket" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:54.523497 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.523467 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-kube-api-access-gdtsc" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "kube-api-access-gdtsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:54.523620 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.523538 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-envoy" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:54.523620 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.523552 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801b2255-648f-4abe-91f6-d29770490815-istio-podinfo" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:09:54.523620 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.523541 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-istio-token" pod "801b2255-648f-4abe-91f6-d29770490815" (UID: "801b2255-648f-4abe-91f6-d29770490815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:54.622627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.622594 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.622627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.622622 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.622627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.622632 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b2255-648f-4abe-91f6-d29770490815-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.622854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.622641 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b2255-648f-4abe-91f6-d29770490815-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.622854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.622650 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdtsc\" (UniqueName: \"kubernetes.io/projected/801b2255-648f-4abe-91f6-d29770490815-kube-api-access-gdtsc\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.711717 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.711673 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6"] Nov 02 22:09:54.715125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.715095 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj"] Nov 02 22:09:54.816921 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:54.816893 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:55.298770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.298678 4441 scope.go:122] "RemoveContainer" containerID="1d5072f94f5346fbb585d8c6841e8764b8baa82e9879304b2e7ba7ca7fd4a92c" Nov 02 22:09:55.308350 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.308327 4441 scope.go:122] "RemoveContainer" containerID="4c00219cd4fa9c1647445ceb190b25981255c0b09d2ba0a319eb7cb6df54677f" Nov 02 22:09:55.317158 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.317140 4441 scope.go:122] "RemoveContainer" containerID="a71b7d1785c818e5e2805e252bb1446d5c60c2c395fe5778904f4c1517ccf484" Nov 02 22:09:55.324929 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.324910 4441 scope.go:122] "RemoveContainer" containerID="fe92ccb99ab16a0f40b9dcf5d06c5144da609e66a3e56898aec8cdcbc525bdd6" Nov 02 22:09:55.332836 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.332818 4441 scope.go:122] "RemoveContainer" containerID="1fed1e1c5f0903b40e12e8b3d2aa867b4b2628677ae962c344dd367334932323" Nov 02 22:09:55.340191 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.340176 4441 scope.go:122] "RemoveContainer" containerID="b5630e85a024da68aaf1bdebcfeba83fe3ab5bbf2dd799d96fea6ffc55b8af82" Nov 02 22:09:55.358130 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.358072 4441 generic.go:356] "Generic (PLEG): container finished" podID="801b2255-648f-4abe-91f6-d29770490815" containerID="57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42" exitCode=0 Nov 02 22:09:55.358194 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.358135 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" Nov 02 22:09:55.358194 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.358128 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" event={"ID":"801b2255-648f-4abe-91f6-d29770490815","Type":"ContainerDied","Data":"57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42"} Nov 02 22:09:55.358303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.358199 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv" event={"ID":"801b2255-648f-4abe-91f6-d29770490815","Type":"ContainerDied","Data":"19aff9c0912db8b95a2c7192b72b3229d0468f47d4825f2bbbbcb54dc646e839"} Nov 02 22:09:55.358303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.358236 4441 scope.go:122] "RemoveContainer" containerID="57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42" Nov 02 22:09:55.366676 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.366638 4441 scope.go:122] "RemoveContainer" containerID="57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42" Nov 02 22:09:55.366887 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:55.366865 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42\": container with ID starting with 57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42 not found: ID does not exist" containerID="57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42" Nov 02 22:09:55.366962 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.366897 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42"} err="failed to get container status \"57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42\": rpc error: code = NotFound desc = could not find container \"57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42\": container with ID starting with 57aa88b6100711ce672cc1f9ec9eb10adf22294c5bb466ef5643b05e2092fd42 not found: ID does not exist" Nov 02 22:09:55.574259 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.574232 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv"] Nov 02 22:09:55.579058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:55.579036 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-49c06d-openshift-default-795bb8bd77-pgndv"] Nov 02 22:09:56.583635 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:56.583604 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801b2255-648f-4abe-91f6-d29770490815" path="/var/lib/kubelet/pods/801b2255-648f-4abe-91f6-d29770490815/volumes" Nov 02 22:09:56.771063 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:56.771027 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:57.041658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.041577 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62"] Nov 02 22:09:57.090319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.090285 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f859c-mkm62"] Nov 02 22:09:57.813759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.813730 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:09:57.854467 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.854434 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-plugins-config-volume\") pod \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " Nov 02 22:09:57.854631 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.854610 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-kube-api-access-sdxf6\" (UniqueName: \"kubernetes.io/projected/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-kube-api-access-sdxf6\") pod \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\" (UID: \"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd\") " Nov 02 22:09:57.854838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.854813 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-plugins-config-volume" pod "cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" (UID: "cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:57.854981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.854949 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:57.856636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.856608 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-kube-api-access-sdxf6" pod "cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" (UID: "cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd"). InnerVolumeSpecName "kube-api-access-sdxf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:57.893779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.893760 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:09:57.956204 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.956173 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150790a2-98b0-44ba-ab92-06ab0a0311dd-kube-api-access-zz8h4\" (UniqueName: \"kubernetes.io/projected/150790a2-98b0-44ba-ab92-06ab0a0311dd-kube-api-access-zz8h4\") pod \"150790a2-98b0-44ba-ab92-06ab0a0311dd\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " Nov 02 22:09:57.956352 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.956243 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150790a2-98b0-44ba-ab92-06ab0a0311dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150790a2-98b0-44ba-ab92-06ab0a0311dd-plugins-config-volume\") pod \"150790a2-98b0-44ba-ab92-06ab0a0311dd\" (UID: \"150790a2-98b0-44ba-ab92-06ab0a0311dd\") " Nov 02 22:09:57.956664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.956636 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxf6\" (UniqueName: \"kubernetes.io/projected/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd-kube-api-access-sdxf6\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:57.956769 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.956713 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150790a2-98b0-44ba-ab92-06ab0a0311dd-plugins-config-volume" pod "150790a2-98b0-44ba-ab92-06ab0a0311dd" (UID: "150790a2-98b0-44ba-ab92-06ab0a0311dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:57.958221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:57.958203 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150790a2-98b0-44ba-ab92-06ab0a0311dd-kube-api-access-zz8h4" pod "150790a2-98b0-44ba-ab92-06ab0a0311dd" (UID: "150790a2-98b0-44ba-ab92-06ab0a0311dd"). InnerVolumeSpecName "kube-api-access-zz8h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:58.044275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.044242 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m"] Nov 02 22:09:58.046564 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.046525 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-773e8060-g284m"] Nov 02 22:09:58.057788 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.057764 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150790a2-98b0-44ba-ab92-06ab0a0311dd-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:58.057788 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.057787 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz8h4\" (UniqueName: \"kubernetes.io/projected/150790a2-98b0-44ba-ab92-06ab0a0311dd-kube-api-access-zz8h4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:58.381282 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.381249 4441 generic.go:356] "Generic (PLEG): container finished" podID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" containerID="a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad" exitCode=0 Nov 02 22:09:58.381451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.381312 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" Nov 02 22:09:58.381451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.381334 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" event={"ID":"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd","Type":"ContainerDied","Data":"a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad"} Nov 02 22:09:58.381451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.381421 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v" event={"ID":"cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd","Type":"ContainerDied","Data":"29181a07394ca1856404329c68f76185be2c18d4048d13608923bcc58af5ecdc"} Nov 02 22:09:58.381451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.381450 4441 scope.go:122] "RemoveContainer" containerID="a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad" Nov 02 22:09:58.382829 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.382797 4441 generic.go:356] "Generic (PLEG): container finished" podID="150790a2-98b0-44ba-ab92-06ab0a0311dd" containerID="74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4" exitCode=0 Nov 02 22:09:58.382999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.382849 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" Nov 02 22:09:58.382999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.382884 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" event={"ID":"150790a2-98b0-44ba-ab92-06ab0a0311dd","Type":"ContainerDied","Data":"74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4"} Nov 02 22:09:58.382999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.382938 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf" event={"ID":"150790a2-98b0-44ba-ab92-06ab0a0311dd","Type":"ContainerDied","Data":"a622365c7ddc4077de389619a5bfa0815c487d5bb438fc4f82364828e8105176"} Nov 02 22:09:58.392585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.392560 4441 scope.go:122] "RemoveContainer" containerID="a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad" Nov 02 22:09:58.392925 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:58.392841 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad\": container with ID starting with a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad not found: ID does not exist" containerID="a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad" Nov 02 22:09:58.392925 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.392868 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad"} err="failed to get container status \"a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad\": rpc error: code = NotFound desc = could not find container \"a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad\": container with ID starting with a5e768502bc403b635bac816b55556a0e51d601db9053fa03e13fd32414b3aad not found: ID does not exist" Nov 02 22:09:58.392925 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.392885 4441 scope.go:122] "RemoveContainer" containerID="74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4" Nov 02 22:09:58.401366 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.401351 4441 scope.go:122] "RemoveContainer" containerID="74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4" Nov 02 22:09:58.401614 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:09:58.401597 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4\": container with ID starting with 74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4 not found: ID does not exist" containerID="74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4" Nov 02 22:09:58.401678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.401620 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4"} err="failed to get container status \"74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4\": rpc error: code = NotFound desc = could not find container \"74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4\": container with ID starting with 74b2a6643c376c4bb0f2a7d7893c1656ece8c3a4f1c2f7b77a517a62563b65d4 not found: ID does not exist" Nov 02 22:09:58.536677 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.536641 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf"] Nov 02 22:09:58.540193 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.540165 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49c06d-67bcf7b5c9-l2spf"] Nov 02 22:09:58.583353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.583319 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d746677-71a0-4ebb-ba09-b45fa261e8d6" path="/var/lib/kubelet/pods/0d746677-71a0-4ebb-ba09-b45fa261e8d6/volumes" Nov 02 22:09:58.583753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.583736 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150790a2-98b0-44ba-ab92-06ab0a0311dd" path="/var/lib/kubelet/pods/150790a2-98b0-44ba-ab92-06ab0a0311dd/volumes" Nov 02 22:09:58.584155 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.584141 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab1d0b4-7423-4644-b4e6-21dedcd645ca" path="/var/lib/kubelet/pods/fab1d0b4-7423-4644-b4e6-21dedcd645ca/volumes" Nov 02 22:09:58.642075 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.642021 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v"] Nov 02 22:09:58.647720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:58.647700 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49c06d-85d8b67ff5-lks6v"] Nov 02 22:09:59.688985 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:09:59.688936 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:00.581540 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.581508 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" path="/var/lib/kubelet/pods/cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd/volumes" Nov 02 22:10:00.821223 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821186 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42"] Nov 02 22:10:00.821727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821705 4441 cpu_manager.go:414] "containerMap: removing container" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" containerName="payload-pre-processing" Nov 02 22:10:00.821820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821729 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" containerName="payload-pre-processing" Nov 02 22:10:00.821820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821746 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" Nov 02 22:10:00.821820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821754 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" Nov 02 22:10:00.821820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821767 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6f51f46e-d646-446b-bf06-b38a5c4b53ef" containerName="revoke-keys" Nov 02 22:10:00.821820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821774 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f51f46e-d646-446b-bf06-b38a5c4b53ef" containerName="revoke-keys" Nov 02 22:10:00.821820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821808 4441 cpu_manager.go:414] "containerMap: removing container" podUID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" containerName="payload-processing" Nov 02 22:10:00.821820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821815 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" containerName="payload-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821834 4441 cpu_manager.go:414] "containerMap: removing container" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821842 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821853 4441 cpu_manager.go:414] "containerMap: removing container" podUID="150790a2-98b0-44ba-ab92-06ab0a0311dd" containerName="payload-pre-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821860 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150790a2-98b0-44ba-ab92-06ab0a0311dd" containerName="payload-pre-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821873 4441 cpu_manager.go:414] "containerMap: removing container" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821880 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821892 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2f516506-1916-4436-bc71-7da08d430134" containerName="payload-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821899 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f516506-1916-4436-bc71-7da08d430134" containerName="payload-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821911 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.821918 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822038 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f51f46e-d646-446b-bf06-b38a5c4b53ef" containerName="revoke-keys" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822058 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc03a9eb-98bb-48dd-b05b-21d3f8d43ccd" containerName="payload-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822070 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822079 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="150790a2-98b0-44ba-ab92-06ab0a0311dd" containerName="payload-pre-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822087 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="801b2255-648f-4abe-91f6-d29770490815" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822097 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f516506-1916-4436-bc71-7da08d430134" containerName="payload-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822107 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822119 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="29544df9-e93d-43fd-93fc-dacbc538008a" containerName="payload-pre-processing" Nov 02 22:10:00.822235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.822128 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aac1cb0-9792-4d93-b66e-259a1d6a6500" containerName="istio-proxy" Nov 02 22:10:00.826999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.826956 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.876833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.876809 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:00.876953 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.876896 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:00.882339 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.882320 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.882400 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.882373 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8v86\" (UniqueName: \"kubernetes.io/projected/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-kube-api-access-k8v86\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.882477 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.882465 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.924217 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.924186 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:00.924346 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.924253 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:10:00.928456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.928432 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42"] Nov 02 22:10:00.983922 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.983891 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8v86\" (UniqueName: \"kubernetes.io/projected/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-kube-api-access-k8v86\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.984079 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.983939 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.984079 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.984041 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.984702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.984680 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:00.986268 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:00.986244 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:01.037262 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:01.037231 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8v86\" (UniqueName: \"kubernetes.io/projected/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-kube-api-access-k8v86\") pod \"maas-api-revoke-sub-171072d1-gfw42\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:01.140157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:01.140070 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:01.504131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:01.504101 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42"] Nov 02 22:10:01.505437 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:10:01.505408 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bf94db_222d_4d4a_ac2b_c0024935d7ca.slice/crio-d8ca7cae13364e82a52c369f44dbf57d4bdb1c74d4535d16412913f949e1d145 WatchSource:0}: Error finding container d8ca7cae13364e82a52c369f44dbf57d4bdb1c74d4535d16412913f949e1d145: Status 404 returned error can't find the container with id d8ca7cae13364e82a52c369f44dbf57d4bdb1c74d4535d16412913f949e1d145 Nov 02 22:10:01.766388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:01.766322 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk"] Nov 02 22:10:02.005887 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:02.005858 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:02.120170 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:02.120136 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:02.409835 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:02.409802 4441 generic.go:356] "Generic (PLEG): container finished" podID="b1bf94db-222d-4d4a-ac2b-c0024935d7ca" containerID="b8d5b5265ac5a6300a8ae323735cf8a85f1ce59b4c034d7a5afe820d0756cc89" exitCode=0 Nov 02 22:10:02.409989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:02.409882 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" event={"ID":"b1bf94db-222d-4d4a-ac2b-c0024935d7ca","Type":"ContainerDied","Data":"b8d5b5265ac5a6300a8ae323735cf8a85f1ce59b4c034d7a5afe820d0756cc89"} Nov 02 22:10:02.409989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:02.409932 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" event={"ID":"b1bf94db-222d-4d4a-ac2b-c0024935d7ca","Type":"ContainerStarted","Data":"d8ca7cae13364e82a52c369f44dbf57d4bdb1c74d4535d16412913f949e1d145"} Nov 02 22:10:03.548945 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.548917 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:03.605029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.604993 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-xks-ca\") pod \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " Nov 02 22:10:03.605201 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.605057 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-openshift-ca\") pod \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " Nov 02 22:10:03.605298 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.605220 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-kube-api-access-k8v86\" (UniqueName: \"kubernetes.io/projected/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-kube-api-access-k8v86\") pod \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\" (UID: \"b1bf94db-222d-4d4a-ac2b-c0024935d7ca\") " Nov 02 22:10:03.605463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.605441 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-openshift-ca" pod "b1bf94db-222d-4d4a-ac2b-c0024935d7ca" (UID: "b1bf94db-222d-4d4a-ac2b-c0024935d7ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:03.607063 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.607030 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-xks-ca" pod "b1bf94db-222d-4d4a-ac2b-c0024935d7ca" (UID: "b1bf94db-222d-4d4a-ac2b-c0024935d7ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:03.607228 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.607210 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-kube-api-access-k8v86" pod "b1bf94db-222d-4d4a-ac2b-c0024935d7ca" (UID: "b1bf94db-222d-4d4a-ac2b-c0024935d7ca"). InnerVolumeSpecName "kube-api-access-k8v86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:03.707086 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.706992 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8v86\" (UniqueName: \"kubernetes.io/projected/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-kube-api-access-k8v86\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:03.707086 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.707031 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:03.707086 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:03.707047 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bf94db-222d-4d4a-ac2b-c0024935d7ca-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:04.423876 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:04.423845 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" Nov 02 22:10:04.423876 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:04.423857 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42" event={"ID":"b1bf94db-222d-4d4a-ac2b-c0024935d7ca","Type":"ContainerDied","Data":"d8ca7cae13364e82a52c369f44dbf57d4bdb1c74d4535d16412913f949e1d145"} Nov 02 22:10:04.424116 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:04.423911 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8ca7cae13364e82a52c369f44dbf57d4bdb1c74d4535d16412913f949e1d145" Nov 02 22:10:06.766743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:06.766700 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" podUID="2d4c566e-055c-494d-8299-d0871f6e6efc" containerName="payload-pre-processing" containerID="cri-o://b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e" gracePeriod=25 Nov 02 22:10:09.732193 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:09.732154 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" containerID="cri-o://1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6" gracePeriod=45 Nov 02 22:10:09.734100 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:09.734052 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" containerID="cri-o://a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18" gracePeriod=45 Nov 02 22:10:09.738418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:09.738382 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": EOF" Nov 02 22:10:09.989987 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:09.989912 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 22:10:10.012881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.012854 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 22:10:10.065264 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065238 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36377f3c-6f41-4ce3-8778-59f87630e3ab-kube-api-access-hdsfp\" (UniqueName: \"kubernetes.io/projected/36377f3c-6f41-4ce3-8778-59f87630e3ab-kube-api-access-hdsfp\") pod \"36377f3c-6f41-4ce3-8778-59f87630e3ab\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " Nov 02 22:10:10.065389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065280 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36377f3c-6f41-4ce3-8778-59f87630e3ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/36377f3c-6f41-4ce3-8778-59f87630e3ab-tls-certs\") pod \"36377f3c-6f41-4ce3-8778-59f87630e3ab\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " Nov 02 22:10:10.065389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065333 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-home\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-home\") pod \"93c7eea9-bba3-4e14-9702-3c837a91d97d\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " Nov 02 22:10:10.065389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065369 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-home\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-home\") pod \"36377f3c-6f41-4ce3-8778-59f87630e3ab\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " Nov 02 22:10:10.065624 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065594 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c7eea9-bba3-4e14-9702-3c837a91d97d-kube-api-access-gk4jz\" (UniqueName: \"kubernetes.io/projected/93c7eea9-bba3-4e14-9702-3c837a91d97d-kube-api-access-gk4jz\") pod \"93c7eea9-bba3-4e14-9702-3c837a91d97d\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " Nov 02 22:10:10.065785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065604 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-home" pod "93c7eea9-bba3-4e14-9702-3c837a91d97d" (UID: "93c7eea9-bba3-4e14-9702-3c837a91d97d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.065785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065648 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-tmp-dir\") pod \"36377f3c-6f41-4ce3-8778-59f87630e3ab\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " Nov 02 22:10:10.065785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065610 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-home" pod "36377f3c-6f41-4ce3-8778-59f87630e3ab" (UID: "36377f3c-6f41-4ce3-8778-59f87630e3ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.065785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065705 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-dshm\") pod \"93c7eea9-bba3-4e14-9702-3c837a91d97d\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " Nov 02 22:10:10.065785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065734 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-dshm\") pod \"36377f3c-6f41-4ce3-8778-59f87630e3ab\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " Nov 02 22:10:10.065785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065770 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-model-cache\") pod \"36377f3c-6f41-4ce3-8778-59f87630e3ab\" (UID: \"36377f3c-6f41-4ce3-8778-59f87630e3ab\") " Nov 02 22:10:10.066173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065812 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-model-cache\") pod \"93c7eea9-bba3-4e14-9702-3c837a91d97d\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " Nov 02 22:10:10.066173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065839 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-tmp-dir\") pod \"93c7eea9-bba3-4e14-9702-3c837a91d97d\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " Nov 02 22:10:10.066173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065869 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c7eea9-bba3-4e14-9702-3c837a91d97d-tls-certs\" (UniqueName: \"kubernetes.io/secret/93c7eea9-bba3-4e14-9702-3c837a91d97d-tls-certs\") pod \"93c7eea9-bba3-4e14-9702-3c837a91d97d\" (UID: \"93c7eea9-bba3-4e14-9702-3c837a91d97d\") " Nov 02 22:10:10.066173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.065938 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-tmp-dir" pod "36377f3c-6f41-4ce3-8778-59f87630e3ab" (UID: "36377f3c-6f41-4ce3-8778-59f87630e3ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.066393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.066271 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.066393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.066292 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.066393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.066306 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.066393 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.066332 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-model-cache" pod "36377f3c-6f41-4ce3-8778-59f87630e3ab" (UID: "36377f3c-6f41-4ce3-8778-59f87630e3ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.066629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.066531 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-model-cache" pod "93c7eea9-bba3-4e14-9702-3c837a91d97d" (UID: "93c7eea9-bba3-4e14-9702-3c837a91d97d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.066629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.066542 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-tmp-dir" pod "93c7eea9-bba3-4e14-9702-3c837a91d97d" (UID: "93c7eea9-bba3-4e14-9702-3c837a91d97d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.068031 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.068005 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-dshm" pod "93c7eea9-bba3-4e14-9702-3c837a91d97d" (UID: "93c7eea9-bba3-4e14-9702-3c837a91d97d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.068169 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.068147 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36377f3c-6f41-4ce3-8778-59f87630e3ab-tls-certs" pod "36377f3c-6f41-4ce3-8778-59f87630e3ab" (UID: "36377f3c-6f41-4ce3-8778-59f87630e3ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:10.068247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.068163 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36377f3c-6f41-4ce3-8778-59f87630e3ab-kube-api-access-hdsfp" pod "36377f3c-6f41-4ce3-8778-59f87630e3ab" (UID: "36377f3c-6f41-4ce3-8778-59f87630e3ab"). InnerVolumeSpecName "kube-api-access-hdsfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:10.068451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.068436 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c7eea9-bba3-4e14-9702-3c837a91d97d-kube-api-access-gk4jz" pod "93c7eea9-bba3-4e14-9702-3c837a91d97d" (UID: "93c7eea9-bba3-4e14-9702-3c837a91d97d"). InnerVolumeSpecName "kube-api-access-gk4jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:10.068564 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.068546 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-dshm" pod "36377f3c-6f41-4ce3-8778-59f87630e3ab" (UID: "36377f3c-6f41-4ce3-8778-59f87630e3ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.068611 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.068547 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c7eea9-bba3-4e14-9702-3c837a91d97d-tls-certs" pod "93c7eea9-bba3-4e14-9702-3c837a91d97d" (UID: "93c7eea9-bba3-4e14-9702-3c837a91d97d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:10.167035 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167001 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdsfp\" (UniqueName: \"kubernetes.io/projected/36377f3c-6f41-4ce3-8778-59f87630e3ab-kube-api-access-hdsfp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167035 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167030 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36377f3c-6f41-4ce3-8778-59f87630e3ab-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167035 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167041 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4jz\" (UniqueName: \"kubernetes.io/projected/93c7eea9-bba3-4e14-9702-3c837a91d97d-kube-api-access-gk4jz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167050 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167061 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167069 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36377f3c-6f41-4ce3-8778-59f87630e3ab-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167077 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167085 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93c7eea9-bba3-4e14-9702-3c837a91d97d-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.167276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.167093 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93c7eea9-bba3-4e14-9702-3c837a91d97d-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.465587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.465551 4441 generic.go:356] "Generic (PLEG): container finished" podID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerID="a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18" exitCode=0 Nov 02 22:10:10.465806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.465603 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" Nov 02 22:10:10.465806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.465635 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" event={"ID":"93c7eea9-bba3-4e14-9702-3c837a91d97d","Type":"ContainerDied","Data":"a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18"} Nov 02 22:10:10.465806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.465695 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj" event={"ID":"93c7eea9-bba3-4e14-9702-3c837a91d97d","Type":"ContainerDied","Data":"ff0f3b92fe10e9317f91f539a0eec3c6276c8926b12b5019ff8c59368ff27992"} Nov 02 22:10:10.465806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.465720 4441 scope.go:122] "RemoveContainer" containerID="a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18" Nov 02 22:10:10.467223 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.467196 4441 generic.go:356] "Generic (PLEG): container finished" podID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerID="1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6" exitCode=0 Nov 02 22:10:10.467346 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.467231 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" Nov 02 22:10:10.467346 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.467235 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" event={"ID":"36377f3c-6f41-4ce3-8778-59f87630e3ab","Type":"ContainerDied","Data":"1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6"} Nov 02 22:10:10.467346 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.467260 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6" event={"ID":"36377f3c-6f41-4ce3-8778-59f87630e3ab","Type":"ContainerDied","Data":"39959a205ad95e537f6c61593a7eec7280c9f422879a4e0063391a502e27695c"} Nov 02 22:10:10.476444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.476424 4441 scope.go:122] "RemoveContainer" containerID="a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18" Nov 02 22:10:10.476738 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:10:10.476719 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18\": container with ID starting with a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18 not found: ID does not exist" containerID="a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18" Nov 02 22:10:10.476817 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.476751 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18"} err="failed to get container status \"a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18\": rpc error: code = NotFound desc = could not find container \"a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18\": container with ID starting with a0a6ba3e8928aaafe39e26a7c1a9552173c71e50dc87b70497aa7aea552a5b18 not found: ID does not exist" Nov 02 22:10:10.476817 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.476774 4441 scope.go:122] "RemoveContainer" containerID="1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6" Nov 02 22:10:10.485222 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.485205 4441 scope.go:122] "RemoveContainer" containerID="1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6" Nov 02 22:10:10.485469 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:10:10.485451 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6\": container with ID starting with 1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6 not found: ID does not exist" containerID="1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6" Nov 02 22:10:10.485516 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.485478 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6"} err="failed to get container status \"1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6\": rpc error: code = NotFound desc = could not find container \"1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6\": container with ID starting with 1611a9709f078f1a552bce5ca4d63ffd79e16f564736941d1698fbeed54f5ff6 not found: ID does not exist" Nov 02 22:10:10.613415 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:10:10.613383 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:10:10.619909 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.619870 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6"] Nov 02 22:10:10.622248 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.622228 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-2-w6-284b8e-kserve-ccdf8c4c9-rbvr6"] Nov 02 22:10:10.769711 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.769686 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj"] Nov 02 22:10:10.771991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:10.771949 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/premium-w6-284b8e-kserve-76b6b5b6f7-tjqbj"] Nov 02 22:10:12.520984 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:12.520929 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" podUID="2d4c566e-055c-494d-8299-d0871f6e6efc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:10:12.581642 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:12.581614 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" path="/var/lib/kubelet/pods/36377f3c-6f41-4ce3-8778-59f87630e3ab/volumes" Nov 02 22:10:12.582037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:12.582023 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" path="/var/lib/kubelet/pods/93c7eea9-bba3-4e14-9702-3c837a91d97d/volumes" Nov 02 22:10:14.971166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:14.971122 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.395465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:16.395428 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.673412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:16.673315 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:17.316720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:17.316688 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:22.521335 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:22.521293 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" podUID="2d4c566e-055c-494d-8299-d0871f6e6efc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:10:23.082507 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:23.082473 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:23.216548 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:23.216510 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:23.420440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:23.420349 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw"] Nov 02 22:10:23.420633 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:23.420561 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" containerID="cri-o://df7b22466548e2dea8b74c1f9f707a5440ebc7a41aa94fc9b53c31822a4c26d7" gracePeriod=30 Nov 02 22:10:23.426375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:23.424545 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:24.909509 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:24.909474 4441 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:10:24.909980 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:24.909540 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:10:27.007500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.007469 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 22:10:27.120902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.120875 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4c566e-055c-494d-8299-d0871f6e6efc-kube-api-access-w2vj2\" (UniqueName: \"kubernetes.io/projected/2d4c566e-055c-494d-8299-d0871f6e6efc-kube-api-access-w2vj2\") pod \"2d4c566e-055c-494d-8299-d0871f6e6efc\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " Nov 02 22:10:27.121099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.120954 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4c566e-055c-494d-8299-d0871f6e6efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4c566e-055c-494d-8299-d0871f6e6efc-plugins-config-volume\") pod \"2d4c566e-055c-494d-8299-d0871f6e6efc\" (UID: \"2d4c566e-055c-494d-8299-d0871f6e6efc\") " Nov 02 22:10:27.121330 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.121307 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4c566e-055c-494d-8299-d0871f6e6efc-plugins-config-volume" pod "2d4c566e-055c-494d-8299-d0871f6e6efc" (UID: "2d4c566e-055c-494d-8299-d0871f6e6efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:27.123058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.123038 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4c566e-055c-494d-8299-d0871f6e6efc-kube-api-access-w2vj2" pod "2d4c566e-055c-494d-8299-d0871f6e6efc" (UID: "2d4c566e-055c-494d-8299-d0871f6e6efc"). InnerVolumeSpecName "kube-api-access-w2vj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:27.221557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.221530 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2vj2\" (UniqueName: \"kubernetes.io/projected/2d4c566e-055c-494d-8299-d0871f6e6efc-kube-api-access-w2vj2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:27.221557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.221555 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4c566e-055c-494d-8299-d0871f6e6efc-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:27.572587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.572548 4441 generic.go:356] "Generic (PLEG): container finished" podID="2d4c566e-055c-494d-8299-d0871f6e6efc" containerID="b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e" exitCode=0 Nov 02 22:10:27.572771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.572609 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" Nov 02 22:10:27.572771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.572624 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" event={"ID":"2d4c566e-055c-494d-8299-d0871f6e6efc","Type":"ContainerDied","Data":"b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e"} Nov 02 22:10:27.572771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.572680 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk" event={"ID":"2d4c566e-055c-494d-8299-d0871f6e6efc","Type":"ContainerDied","Data":"33f79ec77d020f2e1e0ad3e7a23b16b2188928a1489d2be6a72dc4382c9f64af"} Nov 02 22:10:27.572771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.572709 4441 scope.go:122] "RemoveContainer" containerID="b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e" Nov 02 22:10:27.582444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.582423 4441 scope.go:122] "RemoveContainer" containerID="b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e" Nov 02 22:10:27.582691 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:10:27.582673 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e\": container with ID starting with b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e not found: ID does not exist" containerID="b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e" Nov 02 22:10:27.582738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.582699 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e"} err="failed to get container status \"b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e\": rpc error: code = NotFound desc = could not find container \"b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e\": container with ID starting with b4f13d5f8197da9bd780b9e3ca44435af9edfb35c9993edf8ff4c746ed2cb03e not found: ID does not exist" Nov 02 22:10:27.728831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.728799 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk"] Nov 02 22:10:27.775405 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:27.775381 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-284b8e-548df84bc-wznlk"] Nov 02 22:10:28.582226 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.582192 4441 generic.go:356] "Generic (PLEG): container finished" podID="da9492a3-2097-4035-b212-ae86afc0667b" containerID="df7b22466548e2dea8b74c1f9f707a5440ebc7a41aa94fc9b53c31822a4c26d7" exitCode=0 Nov 02 22:10:28.582828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.582806 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4c566e-055c-494d-8299-d0871f6e6efc" path="/var/lib/kubelet/pods/2d4c566e-055c-494d-8299-d0871f6e6efc/volumes" Nov 02 22:10:28.583171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.583144 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" event={"ID":"da9492a3-2097-4035-b212-ae86afc0667b","Type":"ContainerDied","Data":"df7b22466548e2dea8b74c1f9f707a5440ebc7a41aa94fc9b53c31822a4c26d7"} Nov 02 22:10:28.669482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.669454 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 22:10:28.834627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834600 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9492a3-2097-4035-b212-ae86afc0667b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9492a3-2097-4035-b212-ae86afc0667b-istiod-ca-cert\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.834825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834651 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-socket\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.834825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834673 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-data\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.834825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834787 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-credential-socket\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.834825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834820 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-certs\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834856 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-envoy\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834934 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-kube-api-access-nbtk7\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-kube-api-access-nbtk7\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835018 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da9492a3-2097-4035-b212-ae86afc0667b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9492a3-2097-4035-b212-ae86afc0667b-istio-podinfo\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834933 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-socket" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.834956 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-data" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835061 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-istio-token\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-istio-token\") pod \"da9492a3-2097-4035-b212-ae86afc0667b\" (UID: \"da9492a3-2097-4035-b212-ae86afc0667b\") " Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835100 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-certs" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835120 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-credential-socket" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:28.835131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835127 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9492a3-2097-4035-b212-ae86afc0667b-istiod-ca-cert" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:28.835656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835422 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9492a3-2097-4035-b212-ae86afc0667b-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.835656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835437 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.835656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835447 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.835656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835456 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.835656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.835469 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.837270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.837240 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-kube-api-access-nbtk7" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "kube-api-access-nbtk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:28.837270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.837262 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da9492a3-2097-4035-b212-ae86afc0667b-istio-podinfo" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:10:28.837497 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.837480 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-envoy" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:28.837543 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.837487 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-istio-token" pod "da9492a3-2097-4035-b212-ae86afc0667b" (UID: "da9492a3-2097-4035-b212-ae86afc0667b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:28.936544 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.936515 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9492a3-2097-4035-b212-ae86afc0667b-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.936544 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.936543 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbtk7\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-kube-api-access-nbtk7\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.936749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.936553 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9492a3-2097-4035-b212-ae86afc0667b-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:28.936749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:28.936562 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9492a3-2097-4035-b212-ae86afc0667b-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:29.590342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:29.590300 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" event={"ID":"da9492a3-2097-4035-b212-ae86afc0667b","Type":"ContainerDied","Data":"d102bddf5d8cc15288d279aa879cbd6a45893215dd74c0a9444de22306a8a694"} Nov 02 22:10:29.590342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:29.590326 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw" Nov 02 22:10:29.591102 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:29.590359 4441 scope.go:122] "RemoveContainer" containerID="df7b22466548e2dea8b74c1f9f707a5440ebc7a41aa94fc9b53c31822a4c26d7" Nov 02 22:10:29.793526 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:29.793486 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw"] Nov 02 22:10:29.840425 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:29.840352 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-dz5hw"] Nov 02 22:10:30.420981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.420939 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-8wrdx"] Nov 02 22:10:30.421300 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421287 4441 cpu_manager.go:414] "containerMap: removing container" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" Nov 02 22:10:30.421343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421302 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" Nov 02 22:10:30.421343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421315 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b1bf94db-222d-4d4a-ac2b-c0024935d7ca" containerName="revoke-keys" Nov 02 22:10:30.421343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421320 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1bf94db-222d-4d4a-ac2b-c0024935d7ca" containerName="revoke-keys" Nov 02 22:10:30.421343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421330 4441 cpu_manager.go:414] "containerMap: removing container" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" Nov 02 22:10:30.421343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421337 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421351 4441 cpu_manager.go:414] "containerMap: removing container" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421356 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421367 4441 cpu_manager.go:414] "containerMap: removing container" podUID="2d4c566e-055c-494d-8299-d0871f6e6efc" containerName="payload-pre-processing" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421372 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4c566e-055c-494d-8299-d0871f6e6efc" containerName="payload-pre-processing" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421429 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c7eea9-bba3-4e14-9702-3c837a91d97d" containerName="main" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421439 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1bf94db-222d-4d4a-ac2b-c0024935d7ca" containerName="revoke-keys" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421444 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9492a3-2097-4035-b212-ae86afc0667b" containerName="istio-proxy" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421451 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="36377f3c-6f41-4ce3-8778-59f87630e3ab" containerName="main" Nov 02 22:10:30.421505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.421456 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4c566e-055c-494d-8299-d0871f6e6efc" containerName="payload-pre-processing" Nov 02 22:10:30.425756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.425737 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.475845 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.475821 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:10:30.476130 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.476108 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z6pqx\"" Nov 02 22:10:30.476228 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.476155 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:10:30.476228 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.476177 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:10:30.527370 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.527339 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-8wrdx"] Nov 02 22:10:30.548735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.548692 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-metrics-certs\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.548909 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.548774 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-webhook-cert\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.549013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.548978 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczdx\" (UniqueName: \"kubernetes.io/projected/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-kube-api-access-xczdx\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.582934 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.582905 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9492a3-2097-4035-b212-ae86afc0667b" path="/var/lib/kubelet/pods/da9492a3-2097-4035-b212-ae86afc0667b/volumes" Nov 02 22:10:30.650133 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.650095 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xczdx\" (UniqueName: \"kubernetes.io/projected/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-kube-api-access-xczdx\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.650491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.650192 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-metrics-certs\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.650491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.650242 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-webhook-cert\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.652812 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.652784 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-metrics-certs\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.652926 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.652848 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-webhook-cert\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.704175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.704115 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczdx\" (UniqueName: \"kubernetes.io/projected/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-kube-api-access-xczdx\") pod \"maas-controller-9f8dffcbc-8wrdx\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:30.738955 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:30.738933 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:31.106191 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:31.106162 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-8wrdx"] Nov 02 22:10:31.107017 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:10:31.106992 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda084ea1b_c0a9_43fb_90af_8b76ff3da5f4.slice/crio-8fee49d1f92c5c1540ac0858a1b744a3266c41ae5fd38aef14dc49f46dd658d5 WatchSource:0}: Error finding container 8fee49d1f92c5c1540ac0858a1b744a3266c41ae5fd38aef14dc49f46dd658d5: Status 404 returned error can't find the container with id 8fee49d1f92c5c1540ac0858a1b744a3266c41ae5fd38aef14dc49f46dd658d5 Nov 02 22:10:31.108353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:31.108333 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:10:31.605154 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:31.605122 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" event={"ID":"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4","Type":"ContainerStarted","Data":"8fee49d1f92c5c1540ac0858a1b744a3266c41ae5fd38aef14dc49f46dd658d5"} Nov 02 22:10:32.054176 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:32.054061 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:32.207458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:32.207426 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:32.328184 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:32.328145 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:33.620407 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:33.620370 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" event={"ID":"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4","Type":"ContainerStarted","Data":"5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c"} Nov 02 22:10:33.620801 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:33.620506 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:33.731017 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:33.730913 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" podStartSLOduration=1.4418040140000001 podStartE2EDuration="3.730899172s" totalImagesPullingTime="2.289095158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.727828239 +0000 UTC m=+3463.283456649" watchObservedRunningTime="2026-11-02 22:10:33.730899172 +0000 UTC m=+3463.286527526" Nov 02 22:10:37.705710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:37.705666 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:40.616276 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:10:40.616239 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:10:44.628875 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:44.628842 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:10:50.581645 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:50.581599 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:54.935051 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:54.935002 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc"] Nov 02 22:10:54.938884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:54.938861 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:54.989806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:54.989773 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc"] Nov 02 22:10:55.033013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.032962 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:55.033217 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.033026 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:10:55.033217 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.033025 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:55.035475 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.035457 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:55.052942 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.052913 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wp2b\" (UniqueName: \"kubernetes.io/projected/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-kube-api-access-4wp2b\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.053079 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.052950 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.053124 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.053082 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.154128 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.154096 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wp2b\" (UniqueName: \"kubernetes.io/projected/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-kube-api-access-4wp2b\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.154128 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.154130 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.154325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.154173 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.154894 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.154875 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.156617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.156591 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.207740 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.207668 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wp2b\" (UniqueName: \"kubernetes.io/projected/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-kube-api-access-4wp2b\") pod \"maas-api-revoke-sub-85237e45-wjncc\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.252027 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.251997 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:55.404630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.404596 4441 scope.go:122] "RemoveContainer" containerID="b9d0310f8bbe900ca8dfb67317feba294164d0b3637a46903aef0e5e25e5a898" Nov 02 22:10:55.413727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.413707 4441 scope.go:122] "RemoveContainer" containerID="b7f2cb81764c8ccd46d4c43736486434be71768d5801a56561d9792454be48b5" Nov 02 22:10:55.422232 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.422212 4441 scope.go:122] "RemoveContainer" containerID="c9db58053a755458779097c5ab6f28381d89c2aed3386533cb3499244f36f9b6" Nov 02 22:10:55.430731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.430709 4441 scope.go:122] "RemoveContainer" containerID="a1acbe8f8013c52c0b173594be7bac44a4015e082b342149cbf8532301474595" Nov 02 22:10:55.439132 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.439104 4441 scope.go:122] "RemoveContainer" containerID="99237213e465b23d4e8e72a60ba5d4dd602de47c28119a38fe5e352cb7fed89f" Nov 02 22:10:55.446804 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.446788 4441 scope.go:122] "RemoveContainer" containerID="ee1cd48fa809c5e4c9a286b276d121dd8fe627f544501c0a5a67107d8df35bc4" Nov 02 22:10:55.454525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.454505 4441 scope.go:122] "RemoveContainer" containerID="f67c613940cf242e8c50cdaa9dcecb3887c4c62820bd7c2163c266803b5bb899" Nov 02 22:10:55.463063 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.463041 4441 scope.go:122] "RemoveContainer" containerID="47835429f18246970e677a271be2f5c18dda1b32ec7e43c80b94a31845e42782" Nov 02 22:10:55.470487 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.470471 4441 scope.go:122] "RemoveContainer" containerID="3d89d086e4f8c6d197e7d092074b8598c706f2fa24dfaefd8d10accabd0f9a28" Nov 02 22:10:55.614103 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.614078 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc"] Nov 02 22:10:55.615391 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:10:55.615353 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d5a06a_2b7a_4fe9_b90a_531b82a970aa.slice/crio-04fc69016044355b3e59056c53b38cd401bd62220c408f98da5cb1a913fbcbd7 WatchSource:0}: Error finding container 04fc69016044355b3e59056c53b38cd401bd62220c408f98da5cb1a913fbcbd7: Status 404 returned error can't find the container with id 04fc69016044355b3e59056c53b38cd401bd62220c408f98da5cb1a913fbcbd7 Nov 02 22:10:55.744174 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:55.744086 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" event={"ID":"16d5a06a-2b7a-4fe9-b90a-531b82a970aa","Type":"ContainerStarted","Data":"04fc69016044355b3e59056c53b38cd401bd62220c408f98da5cb1a913fbcbd7"} Nov 02 22:10:56.500113 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.500082 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9"] Nov 02 22:10:56.503552 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.503531 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.553859 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.553835 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-445mt\"" Nov 02 22:10:56.567186 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567165 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567271 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567202 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b918a37d-859a-4630-a865-e92121ba6073-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567271 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567229 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxwj\" (UniqueName: \"kubernetes.io/projected/b918a37d-859a-4630-a865-e92121ba6073-kube-api-access-cfxwj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567349 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567280 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b918a37d-859a-4630-a865-e92121ba6073-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567349 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567327 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567423 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567347 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567423 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567418 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b918a37d-859a-4630-a865-e92121ba6073-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567436 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.567490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.567457 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.604508 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.604445 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9"] Nov 02 22:10:56.668133 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668108 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668155 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b918a37d-859a-4630-a865-e92121ba6073-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668195 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxwj\" (UniqueName: \"kubernetes.io/projected/b918a37d-859a-4630-a865-e92121ba6073-kube-api-access-cfxwj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668222 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b918a37d-859a-4630-a865-e92121ba6073-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668319 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668372 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668409 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b918a37d-859a-4630-a865-e92121ba6073-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668469 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668509 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668781 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668597 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668781 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668692 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668781 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668721 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.668905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.668885 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.669090 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.669066 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b918a37d-859a-4630-a865-e92121ba6073-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.670731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.670708 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b918a37d-859a-4630-a865-e92121ba6073-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.671139 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.671121 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b918a37d-859a-4630-a865-e92121ba6073-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.722104 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.722077 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b918a37d-859a-4630-a865-e92121ba6073-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.722316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.722293 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxwj\" (UniqueName: \"kubernetes.io/projected/b918a37d-859a-4630-a865-e92121ba6073-kube-api-access-cfxwj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zt7w9\" (UID: \"b918a37d-859a-4630-a865-e92121ba6073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:56.752370 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.752340 4441 generic.go:356] "Generic (PLEG): container finished" podID="16d5a06a-2b7a-4fe9-b90a-531b82a970aa" containerID="3ae6295bf567b22ab779f52a0cb924267bf762639aef336ad945062b730045f9" exitCode=0 Nov 02 22:10:56.752494 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.752427 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" event={"ID":"16d5a06a-2b7a-4fe9-b90a-531b82a970aa","Type":"ContainerDied","Data":"3ae6295bf567b22ab779f52a0cb924267bf762639aef336ad945062b730045f9"} Nov 02 22:10:56.817128 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:56.817109 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:57.228436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.228401 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9"] Nov 02 22:10:57.229684 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:10:57.229639 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb918a37d_859a_4630_a865_e92121ba6073.slice/crio-63d4def800285ef27ee17f72b05ef55006a5c90b93a99fe6cf594f2b2705a3f2 WatchSource:0}: Error finding container 63d4def800285ef27ee17f72b05ef55006a5c90b93a99fe6cf594f2b2705a3f2: Status 404 returned error can't find the container with id 63d4def800285ef27ee17f72b05ef55006a5c90b93a99fe6cf594f2b2705a3f2 Nov 02 22:10:57.232247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.232210 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:57.232351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.232294 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:57.232351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.232337 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:57.759628 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.759589 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" event={"ID":"b918a37d-859a-4630-a865-e92121ba6073","Type":"ContainerStarted","Data":"bba98c44d84ca40f6ea6209164442a68136affe9bf42407af2808061f39802c7"} Nov 02 22:10:57.760123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.759657 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" event={"ID":"b918a37d-859a-4630-a865-e92121ba6073","Type":"ContainerStarted","Data":"63d4def800285ef27ee17f72b05ef55006a5c90b93a99fe6cf594f2b2705a3f2"} Nov 02 22:10:57.817671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.817631 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:10:57.893013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.889631 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podStartSLOduration=1.8896134020000002 podStartE2EDuration="1.889613402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:57.889376993 +0000 UTC m=+3487.445005331" watchObservedRunningTime="2026-11-02 22:10:57.889613402 +0000 UTC m=+3487.445241741" Nov 02 22:10:57.929377 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.929354 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:57.999549 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.999517 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-xks-ca\") pod \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " Nov 02 22:10:57.999764 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.999566 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-openshift-ca\") pod \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " Nov 02 22:10:57.999764 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.999591 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-kube-api-access-4wp2b\" (UniqueName: \"kubernetes.io/projected/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-kube-api-access-4wp2b\") pod \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\" (UID: \"16d5a06a-2b7a-4fe9-b90a-531b82a970aa\") " Nov 02 22:10:57.999929 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:57.999903 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-openshift-ca" pod "16d5a06a-2b7a-4fe9-b90a-531b82a970aa" (UID: "16d5a06a-2b7a-4fe9-b90a-531b82a970aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:58.001834 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.001805 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-xks-ca" pod "16d5a06a-2b7a-4fe9-b90a-531b82a970aa" (UID: "16d5a06a-2b7a-4fe9-b90a-531b82a970aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:58.001953 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.001842 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-kube-api-access-4wp2b" pod "16d5a06a-2b7a-4fe9-b90a-531b82a970aa" (UID: "16d5a06a-2b7a-4fe9-b90a-531b82a970aa"). InnerVolumeSpecName "kube-api-access-4wp2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:58.100456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.100415 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:58.100456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.100451 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:58.100456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.100464 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wp2b\" (UniqueName: \"kubernetes.io/projected/16d5a06a-2b7a-4fe9-b90a-531b82a970aa-kube-api-access-4wp2b\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:58.766383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.766346 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" Nov 02 22:10:58.766855 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.766355 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc" event={"ID":"16d5a06a-2b7a-4fe9-b90a-531b82a970aa","Type":"ContainerDied","Data":"04fc69016044355b3e59056c53b38cd401bd62220c408f98da5cb1a913fbcbd7"} Nov 02 22:10:58.766855 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.766512 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04fc69016044355b3e59056c53b38cd401bd62220c408f98da5cb1a913fbcbd7" Nov 02 22:10:58.818724 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.818687 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 22:10:58.818869 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:58.818744 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 22:10:59.817916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:59.817885 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 22:10:59.818326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:10:59.817935 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 22:11:00.026187 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:00.026152 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:00.818427 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:00.818389 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 22:11:00.818834 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:00.818442 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 22:11:01.818111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:01.818075 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 22:11:01.818287 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:01.818127 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 22:11:02.818087 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:02.818044 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 22:11:02.818500 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:02.818105 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 22:11:03.817780 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:03.817740 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 22:11:03.818021 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:03.817806 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 22:11:05.818747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:05.818699 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:11:05.819176 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:05.818764 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:06.859099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:06.859065 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:11:06.859469 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:06.859132 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:07.184376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.184296 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:07.335803 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.335758 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-8wrdx"] Nov 02 22:11:07.336037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.336004 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" podUID="a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" containerName="manager" containerID="cri-o://5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c" gracePeriod=30 Nov 02 22:11:07.615508 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.615484 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:11:07.694355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.694258 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-webhook-cert\") pod \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " Nov 02 22:11:07.694355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.694344 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-metrics-certs\") pod \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " Nov 02 22:11:07.694552 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.694394 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:11:07.694842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.694460 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-kube-api-access-xczdx\" (UniqueName: \"kubernetes.io/projected/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-kube-api-access-xczdx\") pod \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\" (UID: \"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4\") " Nov 02 22:11:07.695352 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.695160 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:11:07.696199 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.696179 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" Nov 02 22:11:07.696822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.696800 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-kube-api-access-xczdx" pod "a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" (UID: "a084ea1b-c0a9-43fb-90af-8b76ff3da5f4"). InnerVolumeSpecName "kube-api-access-xczdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:07.697202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.697185 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-webhook-cert" pod "a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" (UID: "a084ea1b-c0a9-43fb-90af-8b76ff3da5f4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:07.697319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.697305 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-metrics-certs" pod "a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" (UID: "a084ea1b-c0a9-43fb-90af-8b76ff3da5f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:07.796338 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.796303 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xczdx\" (UniqueName: \"kubernetes.io/projected/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-kube-api-access-xczdx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:07.796338 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.796339 4441 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-webhook-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:07.796614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.796357 4441 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4-metrics-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:07.844998 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.844927 4441 generic.go:356] "Generic (PLEG): container finished" podID="a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" containerID="5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c" exitCode=0 Nov 02 22:11:07.845185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.845006 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" event={"ID":"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4","Type":"ContainerDied","Data":"5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c"} Nov 02 22:11:07.845185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.845025 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" Nov 02 22:11:07.845185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.845079 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f8dffcbc-8wrdx" event={"ID":"a084ea1b-c0a9-43fb-90af-8b76ff3da5f4","Type":"ContainerDied","Data":"8fee49d1f92c5c1540ac0858a1b744a3266c41ae5fd38aef14dc49f46dd658d5"} Nov 02 22:11:07.845185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.845117 4441 scope.go:122] "RemoveContainer" containerID="5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c" Nov 02 22:11:07.858144 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.858115 4441 scope.go:122] "RemoveContainer" containerID="5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c" Nov 02 22:11:07.858576 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:11:07.858492 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c\": container with ID starting with 5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c not found: ID does not exist" containerID="5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c" Nov 02 22:11:07.858576 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:07.858540 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c"} err="failed to get container status \"5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c\": rpc error: code = NotFound desc = could not find container \"5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c\": container with ID starting with 5d9530d4318f1fbf03019b64ee4ab50c26cc537b314a5ac1bf8cfadb3e9ae67c not found: ID does not exist" Nov 02 22:11:08.001020 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:08.000926 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-8wrdx"] Nov 02 22:11:08.005034 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:08.004999 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-8wrdx"] Nov 02 22:11:08.581543 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:08.581513 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" path="/var/lib/kubelet/pods/a084ea1b-c0a9-43fb-90af-8b76ff3da5f4/volumes" Nov 02 22:11:09.226304 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.226272 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:09.366352 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.366321 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-95qc8"] Nov 02 22:11:09.366675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.366663 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" containerName="manager" Nov 02 22:11:09.366737 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.366676 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" containerName="manager" Nov 02 22:11:09.366737 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.366685 4441 cpu_manager.go:414] "containerMap: removing container" podUID="16d5a06a-2b7a-4fe9-b90a-531b82a970aa" containerName="revoke-keys" Nov 02 22:11:09.366737 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.366691 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d5a06a-2b7a-4fe9-b90a-531b82a970aa" containerName="revoke-keys" Nov 02 22:11:09.366844 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.366757 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d5a06a-2b7a-4fe9-b90a-531b82a970aa" containerName="revoke-keys" Nov 02 22:11:09.366844 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.366773 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a084ea1b-c0a9-43fb-90af-8b76ff3da5f4" containerName="manager" Nov 02 22:11:09.369588 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.369567 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.466826 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.466800 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z6pqx\"" Nov 02 22:11:09.467038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.466817 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:11:09.467038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.466865 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:11:09.468050 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.468027 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:11:09.469683 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.469660 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-95qc8"] Nov 02 22:11:09.511842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.511792 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8pp\" (UniqueName: \"kubernetes.io/projected/f0776fe1-d7f2-4f43-b0e0-a9495263c354-kube-api-access-lj8pp\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.511842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.511820 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0776fe1-d7f2-4f43-b0e0-a9495263c354-metrics-certs\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.511842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.511841 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0776fe1-d7f2-4f43-b0e0-a9495263c354-webhook-cert\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.612692 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.612663 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8pp\" (UniqueName: \"kubernetes.io/projected/f0776fe1-d7f2-4f43-b0e0-a9495263c354-kube-api-access-lj8pp\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.612795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.612696 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0776fe1-d7f2-4f43-b0e0-a9495263c354-metrics-certs\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.612795 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.612717 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0776fe1-d7f2-4f43-b0e0-a9495263c354-webhook-cert\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.615216 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.615191 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0776fe1-d7f2-4f43-b0e0-a9495263c354-metrics-certs\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.615370 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.615348 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0776fe1-d7f2-4f43-b0e0-a9495263c354-webhook-cert\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.666183 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.666162 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8pp\" (UniqueName: \"kubernetes.io/projected/f0776fe1-d7f2-4f43-b0e0-a9495263c354-kube-api-access-lj8pp\") pod \"maas-controller-9f8dffcbc-95qc8\" (UID: \"f0776fe1-d7f2-4f43-b0e0-a9495263c354\") " pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:09.681014 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:09.680993 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:10.040480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:10.040450 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f8dffcbc-95qc8"] Nov 02 22:11:10.041633 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:11:10.041605 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0776fe1_d7f2_4f43_b0e0_a9495263c354.slice/crio-a56afba82546f2ee74e54643cd22f206dfbe4238057993bc8ae228501e70a76e WatchSource:0}: Error finding container a56afba82546f2ee74e54643cd22f206dfbe4238057993bc8ae228501e70a76e: Status 404 returned error can't find the container with id a56afba82546f2ee74e54643cd22f206dfbe4238057993bc8ae228501e70a76e Nov 02 22:11:10.613924 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:11:10.613889 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:11:10.864009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:10.863911 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f8dffcbc-95qc8" event={"ID":"f0776fe1-d7f2-4f43-b0e0-a9495263c354","Type":"ContainerStarted","Data":"a56afba82546f2ee74e54643cd22f206dfbe4238057993bc8ae228501e70a76e"} Nov 02 22:11:11.870807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:11.870769 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f8dffcbc-95qc8" event={"ID":"f0776fe1-d7f2-4f43-b0e0-a9495263c354","Type":"ContainerStarted","Data":"d7030e15fef232205edcdd941299a960259d5b3be7f82c1c3f761794db400323"} Nov 02 22:11:11.871319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:11.870946 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:22.879285 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:22.879253 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9f8dffcbc-95qc8" Nov 02 22:11:22.988460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:11:22.988410 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9f8dffcbc-95qc8" podStartSLOduration=13.100259114 podStartE2EDuration="13.988397877s" totalImagesPullingTime="888.138763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:11.97984167 +0000 UTC m=+3501.535470008" watchObservedRunningTime="2026-11-02 22:11:22.988397877 +0000 UTC m=+3512.544026214" Nov 02 22:11:40.615055 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:11:40.615014 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:12:00.375790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:00.375731 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:10.613338 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:12:10.613297 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:12:12.057789 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.057752 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9"] Nov 02 22:12:12.060383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.060359 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155311 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155277 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155478 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155328 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c0164-57f8-4885-a5df-30433184dc46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155555 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155425 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155620 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmx5c\" (UniqueName: \"kubernetes.io/projected/f47c0164-57f8-4885-a5df-30433184dc46-kube-api-access-zmx5c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155700 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155670 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47c0164-57f8-4885-a5df-30433184dc46-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155718 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155791 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155744 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155791 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155764 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.155887 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.155863 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c0164-57f8-4885-a5df-30433184dc46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.159850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.159830 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9"] Nov 02 22:12:12.256874 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.256844 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.256882 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmx5c\" (UniqueName: \"kubernetes.io/projected/f47c0164-57f8-4885-a5df-30433184dc46-kube-api-access-zmx5c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.256912 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47c0164-57f8-4885-a5df-30433184dc46-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.256931 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257253 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257112 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257253 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257147 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257253 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257190 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c0164-57f8-4885-a5df-30433184dc46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257253 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257244 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257280 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c0164-57f8-4885-a5df-30433184dc46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257337 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257413 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257470 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257640 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.257920 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.257899 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c0164-57f8-4885-a5df-30433184dc46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.259344 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.259327 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c0164-57f8-4885-a5df-30433184dc46-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.259577 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.259562 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c0164-57f8-4885-a5df-30433184dc46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.310324 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.310262 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47c0164-57f8-4885-a5df-30433184dc46-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.310534 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.310512 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmx5c\" (UniqueName: \"kubernetes.io/projected/f47c0164-57f8-4885-a5df-30433184dc46-kube-api-access-zmx5c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-27kg9\" (UID: \"f47c0164-57f8-4885-a5df-30433184dc46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.372035 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.372011 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:12.784429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.784398 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9"] Nov 02 22:12:12.784665 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:12:12.784642 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47c0164_57f8_4885_a5df_30433184dc46.slice/crio-3b490d0ab41afcdf5d8a1879f331ffe5a6c9abb47bbff376811b2d8fe2fd7641 WatchSource:0}: Error finding container 3b490d0ab41afcdf5d8a1879f331ffe5a6c9abb47bbff376811b2d8fe2fd7641: Status 404 returned error can't find the container with id 3b490d0ab41afcdf5d8a1879f331ffe5a6c9abb47bbff376811b2d8fe2fd7641 Nov 02 22:12:12.786916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.786882 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:12.787006 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.786958 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:12.787052 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:12.787022 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:13.207196 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:13.207154 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" event={"ID":"f47c0164-57f8-4885-a5df-30433184dc46","Type":"ContainerStarted","Data":"5ae1271fcae006a243329a5f4ac85a29804ae171b892dac6741d333e04cd4b15"} Nov 02 22:12:13.207707 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:13.207210 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" event={"ID":"f47c0164-57f8-4885-a5df-30433184dc46","Type":"ContainerStarted","Data":"3b490d0ab41afcdf5d8a1879f331ffe5a6c9abb47bbff376811b2d8fe2fd7641"} Nov 02 22:12:13.328941 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:13.327375 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podStartSLOduration=1.327354777 podStartE2EDuration="1.327354777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:13.324613494 +0000 UTC m=+3562.880241833" watchObservedRunningTime="2026-11-02 22:12:13.327354777 +0000 UTC m=+3562.882983128" Nov 02 22:12:13.374995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:13.372442 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:13.378994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:13.378117 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 22:12:13.378994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:13.378188 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 22:12:14.387993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:14.383313 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 22:12:14.387993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:14.383371 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 22:12:15.377991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:15.374329 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 22:12:15.377991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:15.374389 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 22:12:16.387993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:16.377398 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 22:12:16.387993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:16.377471 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 22:12:17.381417 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:17.381373 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 22:12:17.382425 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:17.382391 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 22:12:18.374800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:18.374345 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.374800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:18.374423 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 22:12:19.373317 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:19.373278 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Nov 02 22:12:19.373525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:19.373361 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Nov 02 22:12:21.373395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:21.373349 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:21.373811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:21.373430 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.415167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:22.415120 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:22.415718 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:22.415193 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.416658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:23.416610 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:23.417192 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:23.416687 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.457168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:24.457120 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:24.457660 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:24.457183 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:25.215367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:25.215333 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:25.215821 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:25.215802 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:25.216557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:25.216536 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" Nov 02 22:12:31.898437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.898398 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph"] Nov 02 22:12:31.901104 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.901081 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:31.971783 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.971746 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:31.971954 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.971806 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6rz\" (UniqueName: \"kubernetes.io/projected/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-kube-api-access-5m6rz\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:31.971954 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.971906 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:31.998243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.998220 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:31.998460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.998436 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:31.998585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.998470 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:31.998585 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:31.998566 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:12:32.001332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.001312 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph"] Nov 02 22:12:32.073056 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.073029 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:32.073209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.073063 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6rz\" (UniqueName: \"kubernetes.io/projected/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-kube-api-access-5m6rz\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:32.073209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.073092 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:32.073669 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.073651 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:32.075351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.075330 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:32.127774 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.127748 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6rz\" (UniqueName: \"kubernetes.io/projected/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-kube-api-access-5m6rz\") pod \"maas-api-revoke-sub-2e1f5b5f-bwbph\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:32.213732 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.213669 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:32.589184 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:32.589156 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph"] Nov 02 22:12:32.589986 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:12:32.589941 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad7b3ce_6375_4f34_b3df_6d1331299f1b.slice/crio-55c2ba6f3e1f626343af13455bd8c31c1698ee152334f3c14974cef15f1f3b78 WatchSource:0}: Error finding container 55c2ba6f3e1f626343af13455bd8c31c1698ee152334f3c14974cef15f1f3b78: Status 404 returned error can't find the container with id 55c2ba6f3e1f626343af13455bd8c31c1698ee152334f3c14974cef15f1f3b78 Nov 02 22:12:33.318295 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:33.318255 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" event={"ID":"7ad7b3ce-6375-4f34-b3df-6d1331299f1b","Type":"ContainerStarted","Data":"6fd6a2ed7c012738609e68cd70df490edcdfa530eab1e5362c145d0dcdd93f1f"} Nov 02 22:12:33.318671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:33.318306 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" event={"ID":"7ad7b3ce-6375-4f34-b3df-6d1331299f1b","Type":"ContainerStarted","Data":"55c2ba6f3e1f626343af13455bd8c31c1698ee152334f3c14974cef15f1f3b78"} Nov 02 22:12:33.426770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:33.426720 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" podStartSLOduration=2.426706564 podStartE2EDuration="2.426706564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:33.425917955 +0000 UTC m=+3582.981546293" watchObservedRunningTime="2026-11-02 22:12:33.426706564 +0000 UTC m=+3582.982334930" Nov 02 22:12:34.324867 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:34.324830 4441 generic.go:356] "Generic (PLEG): container finished" podID="7ad7b3ce-6375-4f34-b3df-6d1331299f1b" containerID="6fd6a2ed7c012738609e68cd70df490edcdfa530eab1e5362c145d0dcdd93f1f" exitCode=0 Nov 02 22:12:34.325299 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:34.324917 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" event={"ID":"7ad7b3ce-6375-4f34-b3df-6d1331299f1b","Type":"ContainerDied","Data":"6fd6a2ed7c012738609e68cd70df490edcdfa530eab1e5362c145d0dcdd93f1f"} Nov 02 22:12:35.458689 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.458659 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:35.502907 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.502874 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-kube-api-access-5m6rz\" (UniqueName: \"kubernetes.io/projected/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-kube-api-access-5m6rz\") pod \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " Nov 02 22:12:35.503073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.503001 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-xks-ca\") pod \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " Nov 02 22:12:35.503073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.503052 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-openshift-ca\") pod \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\" (UID: \"7ad7b3ce-6375-4f34-b3df-6d1331299f1b\") " Nov 02 22:12:35.503414 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.503393 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-openshift-ca" pod "7ad7b3ce-6375-4f34-b3df-6d1331299f1b" (UID: "7ad7b3ce-6375-4f34-b3df-6d1331299f1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:35.504898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.504875 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-xks-ca" pod "7ad7b3ce-6375-4f34-b3df-6d1331299f1b" (UID: "7ad7b3ce-6375-4f34-b3df-6d1331299f1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:35.505145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.505127 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-kube-api-access-5m6rz" pod "7ad7b3ce-6375-4f34-b3df-6d1331299f1b" (UID: "7ad7b3ce-6375-4f34-b3df-6d1331299f1b"). InnerVolumeSpecName "kube-api-access-5m6rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:35.604254 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.604179 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m6rz\" (UniqueName: \"kubernetes.io/projected/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-kube-api-access-5m6rz\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:35.604254 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.604207 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:35.604254 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:35.604217 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7b3ce-6375-4f34-b3df-6d1331299f1b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.337995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:36.337942 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" event={"ID":"7ad7b3ce-6375-4f34-b3df-6d1331299f1b","Type":"ContainerDied","Data":"55c2ba6f3e1f626343af13455bd8c31c1698ee152334f3c14974cef15f1f3b78"} Nov 02 22:12:36.338165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:36.338021 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c2ba6f3e1f626343af13455bd8c31c1698ee152334f3c14974cef15f1f3b78" Nov 02 22:12:36.338165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:36.337954 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph" Nov 02 22:12:40.614272 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:12:40.614233 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:12:41.122664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:41.122624 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:44.212105 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.212069 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk"] Nov 02 22:12:44.212532 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.212427 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7ad7b3ce-6375-4f34-b3df-6d1331299f1b" containerName="revoke-keys" Nov 02 22:12:44.212532 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.212439 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad7b3ce-6375-4f34-b3df-6d1331299f1b" containerName="revoke-keys" Nov 02 22:12:44.212532 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.212516 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad7b3ce-6375-4f34-b3df-6d1331299f1b" containerName="revoke-keys" Nov 02 22:12:44.214853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.214830 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.281421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.281377 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.281626 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.281575 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.281626 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.281621 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk4tj\" (UniqueName: \"kubernetes.io/projected/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-kube-api-access-pk4tj\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.311826 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.311792 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:44.312006 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.311985 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:44.312062 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.311994 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:44.312150 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.312135 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:12:44.314896 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.314878 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk"] Nov 02 22:12:44.382597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.382565 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.382806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.382614 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk4tj\" (UniqueName: \"kubernetes.io/projected/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-kube-api-access-pk4tj\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.382806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.382731 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.383422 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.383400 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.384892 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.384870 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.436742 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.436716 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk4tj\" (UniqueName: \"kubernetes.io/projected/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-kube-api-access-pk4tj\") pod \"maas-api-revoke-sub-ec843018-4v8mk\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.526988 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.526904 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:44.888099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:44.888065 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk"] Nov 02 22:12:44.889474 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:12:44.889443 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe63e54_2dea_4ddf_90b3_1a4a2f6bb763.slice/crio-e28e13db544c9fb3001d463c5822d1ea3925306ed282e4b02ab86977dde0308d WatchSource:0}: Error finding container e28e13db544c9fb3001d463c5822d1ea3925306ed282e4b02ab86977dde0308d: Status 404 returned error can't find the container with id e28e13db544c9fb3001d463c5822d1ea3925306ed282e4b02ab86977dde0308d Nov 02 22:12:45.389473 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:45.389433 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" event={"ID":"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763","Type":"ContainerStarted","Data":"d1ddeca7e0d21c8531960efe3ab4798d08abbb222e54251e94cbb7e2fd216bcf"} Nov 02 22:12:45.389863 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:45.389491 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" event={"ID":"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763","Type":"ContainerStarted","Data":"e28e13db544c9fb3001d463c5822d1ea3925306ed282e4b02ab86977dde0308d"} Nov 02 22:12:45.497283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:45.497187 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" podStartSLOduration=1.4971719669999999 podStartE2EDuration="1.497171967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:45.496023061 +0000 UTC m=+3595.051651398" watchObservedRunningTime="2026-11-02 22:12:45.497171967 +0000 UTC m=+3595.052800303" Nov 02 22:12:46.396664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:46.396630 4441 generic.go:356] "Generic (PLEG): container finished" podID="4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" containerID="d1ddeca7e0d21c8531960efe3ab4798d08abbb222e54251e94cbb7e2fd216bcf" exitCode=0 Nov 02 22:12:46.397059 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:46.396717 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" event={"ID":"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763","Type":"ContainerDied","Data":"d1ddeca7e0d21c8531960efe3ab4798d08abbb222e54251e94cbb7e2fd216bcf"} Nov 02 22:12:47.533179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.533155 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:47.612121 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.612090 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-kube-api-access-pk4tj\" (UniqueName: \"kubernetes.io/projected/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-kube-api-access-pk4tj\") pod \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " Nov 02 22:12:47.612323 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.612194 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-xks-ca\") pod \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " Nov 02 22:12:47.612323 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.612222 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-openshift-ca\") pod \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\" (UID: \"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763\") " Nov 02 22:12:47.612699 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.612670 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-openshift-ca" pod "4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" (UID: "4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:47.614241 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.614210 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-kube-api-access-pk4tj" pod "4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" (UID: "4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763"). InnerVolumeSpecName "kube-api-access-pk4tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:47.614611 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.614593 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-xks-ca" pod "4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" (UID: "4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:47.713851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.713768 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:47.713851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.713800 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:47.713851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:47.713816 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk4tj\" (UniqueName: \"kubernetes.io/projected/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763-kube-api-access-pk4tj\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:48.408802 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:48.408768 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" event={"ID":"4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763","Type":"ContainerDied","Data":"e28e13db544c9fb3001d463c5822d1ea3925306ed282e4b02ab86977dde0308d"} Nov 02 22:12:48.408802 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:48.408791 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk" Nov 02 22:12:48.409030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:48.408816 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e28e13db544c9fb3001d463c5822d1ea3925306ed282e4b02ab86977dde0308d" Nov 02 22:12:51.754724 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:51.754691 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:12:51.760892 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:51.760864 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:12:51.763088 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:51.763067 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:12:51.768371 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:51.768353 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:12:51.773666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:51.773645 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:12:51.779882 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:12:51.779862 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:13:00.727748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:00.727716 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:10.619502 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:13:10.619455 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:13:12.476846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.476810 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d"] Nov 02 22:13:12.477333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.477200 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" containerName="revoke-keys" Nov 02 22:13:12.477333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.477213 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" containerName="revoke-keys" Nov 02 22:13:12.477333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.477276 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" containerName="revoke-keys" Nov 02 22:13:12.480538 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.480516 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.532989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.532951 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533127 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.532997 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533127 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.533021 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533127 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.533075 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533237 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.533191 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkw4j\" (UniqueName: \"kubernetes.io/projected/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-kube-api-access-wkw4j\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533274 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.533243 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533312 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.533298 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.533374 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.533425 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.533400 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.583608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.583577 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d"] Nov 02 22:13:12.637486 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637448 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.637684 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637570 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.637684 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637606 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.637684 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637651 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.637880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637690 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.637880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637721 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.637880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637761 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.637880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637786 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.638164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.637880 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkw4j\" (UniqueName: \"kubernetes.io/projected/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-kube-api-access-wkw4j\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.638514 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.638487 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.639130 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.639104 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.639293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.639265 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.639620 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.639602 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.640179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.640158 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.641700 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.641497 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.641700 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.641673 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.692209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.692178 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.692466 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.692447 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkw4j\" (UniqueName: \"kubernetes.io/projected/39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3-kube-api-access-wkw4j\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hft2d\" (UID: \"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:12.792790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:12.792708 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:13.205274 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.205245 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d"] Nov 02 22:13:13.206656 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:13:13.206627 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39be6ac1_bfaf_40e3_a1cc_cbb158c2b3f3.slice/crio-9b8f9d528f73505ef223ac2500da1faecf57c90ad3f27fc90f6d065f8756346e WatchSource:0}: Error finding container 9b8f9d528f73505ef223ac2500da1faecf57c90ad3f27fc90f6d065f8756346e: Status 404 returned error can't find the container with id 9b8f9d528f73505ef223ac2500da1faecf57c90ad3f27fc90f6d065f8756346e Nov 02 22:13:13.208825 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.208797 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:13.208895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.208857 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:13.208895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.208884 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:13.552537 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.552442 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" event={"ID":"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3","Type":"ContainerStarted","Data":"95f89618bdd507b9cf7ee1fbf73bb5d249f107f58989a42fa1a80fcf4ee9d3f6"} Nov 02 22:13:13.552537 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.552500 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" event={"ID":"39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3","Type":"ContainerStarted","Data":"9b8f9d528f73505ef223ac2500da1faecf57c90ad3f27fc90f6d065f8756346e"} Nov 02 22:13:13.663114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.663062 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podStartSLOduration=1.663044062 podStartE2EDuration="1.663044062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:13.661640477 +0000 UTC m=+3623.217268814" watchObservedRunningTime="2026-11-02 22:13:13.663044062 +0000 UTC m=+3623.218672401" Nov 02 22:13:13.793996 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:13.793078 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:14.795605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:14.795566 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 22:13:14.796069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:14.795623 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 22:13:15.043503 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:15.043468 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn"] Nov 02 22:13:15.096328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:15.096293 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c3724d-czgxn"] Nov 02 22:13:15.794362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:15.794325 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 22:13:15.794602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:15.794384 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 22:13:16.597983 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:16.597936 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f57b51-1eb6-4f06-8f31-f345b032d8dd" path="/var/lib/kubelet/pods/02f57b51-1eb6-4f06-8f31-f345b032d8dd/volumes" Nov 02 22:13:16.794433 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:16.794396 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 22:13:16.794617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:16.794461 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 22:13:17.793996 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:17.793377 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 22:13:17.793996 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:17.793443 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 22:13:18.794381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:18.794338 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 22:13:18.794797 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:18.794398 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 22:13:19.793340 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:19.793302 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 02 22:13:19.793558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:19.793361 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 02 22:13:21.572005 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:21.571937 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:21.572691 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:21.572661 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:21.573209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:21.573188 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" Nov 02 22:13:27.595463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.595423 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr"] Nov 02 22:13:27.599059 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.599034 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.690653 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.690612 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.690820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.690662 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.690820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.690691 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.691180 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.690825 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.691180 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.690867 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7cw6\" (UniqueName: \"kubernetes.io/projected/be8ea2f8-52aa-4e10-bca6-b01563cd940f-kube-api-access-g7cw6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.691180 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.690941 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.691180 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.691029 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.691180 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.691058 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.691180 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.691110 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.698730 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.698704 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr"] Nov 02 22:13:27.792196 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792166 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792201 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792230 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792289 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792341 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792329 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7cw6\" (UniqueName: \"kubernetes.io/projected/be8ea2f8-52aa-4e10-bca6-b01563cd940f-kube-api-access-g7cw6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792359 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792406 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792448 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792501 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792731 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792854 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.792950 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792883 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.793029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.792962 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.793174 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.793151 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.794512 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.794489 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.795010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.794992 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.845978 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.845916 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8ea2f8-52aa-4e10-bca6-b01563cd940f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.846163 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.846143 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7cw6\" (UniqueName: \"kubernetes.io/projected/be8ea2f8-52aa-4e10-bca6-b01563cd940f-kube-api-access-g7cw6\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-js4dr\" (UID: \"be8ea2f8-52aa-4e10-bca6-b01563cd940f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:27.913022 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:27.913004 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:28.042514 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.042485 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks"] Nov 02 22:13:28.045270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.045241 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-815733d1-qqcks"] Nov 02 22:13:28.321922 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.321896 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr"] Nov 02 22:13:28.322602 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:13:28.322574 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8ea2f8_52aa_4e10_bca6_b01563cd940f.slice/crio-daa6f776ac88b54c388348eefe4f0d9f53a85d1c0cb8793b5d6e1d8d9b476ae4 WatchSource:0}: Error finding container daa6f776ac88b54c388348eefe4f0d9f53a85d1c0cb8793b5d6e1d8d9b476ae4: Status 404 returned error can't find the container with id daa6f776ac88b54c388348eefe4f0d9f53a85d1c0cb8793b5d6e1d8d9b476ae4 Nov 02 22:13:28.324774 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.324737 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:28.324886 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.324821 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:28.324886 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.324866 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:28.583605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.583520 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f47e60-71d6-49ac-9720-4bc57343bd3e" path="/var/lib/kubelet/pods/25f47e60-71d6-49ac-9720-4bc57343bd3e/volumes" Nov 02 22:13:28.662579 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.662542 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" event={"ID":"be8ea2f8-52aa-4e10-bca6-b01563cd940f","Type":"ContainerStarted","Data":"52a3dda5de63fba1c14f752ab78c27b59fa1be14f47b45631aa9c0fdb6abbf41"} Nov 02 22:13:28.663058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.662606 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" event={"ID":"be8ea2f8-52aa-4e10-bca6-b01563cd940f","Type":"ContainerStarted","Data":"daa6f776ac88b54c388348eefe4f0d9f53a85d1c0cb8793b5d6e1d8d9b476ae4"} Nov 02 22:13:28.780785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.780716 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podStartSLOduration=1.780693985 podStartE2EDuration="1.780693985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:28.780291741 +0000 UTC m=+3638.335920117" watchObservedRunningTime="2026-11-02 22:13:28.780693985 +0000 UTC m=+3638.336322323" Nov 02 22:13:28.914088 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:28.914007 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:29.707129 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.707093 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs"] Nov 02 22:13:29.710919 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.710897 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.761272 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.761250 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:29.761442 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.761319 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:13:29.808316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.808284 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:13:29.808462 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.808287 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:13:29.811334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.811312 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs"] Nov 02 22:13:29.811681 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.811664 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcq4\" (UniqueName: \"kubernetes.io/projected/39f7d8ee-53f8-490e-b411-682873f59856-kube-api-access-jdcq4\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.811798 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.811784 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.811854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.811818 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.912751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.912718 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.912906 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.912795 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcq4\" (UniqueName: \"kubernetes.io/projected/39f7d8ee-53f8-490e-b411-682873f59856-kube-api-access-jdcq4\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.912906 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.912843 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.913386 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.913365 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.914591 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.914563 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 22:13:29.914702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.914614 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 22:13:29.915268 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.915248 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:29.966355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:29.966300 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcq4\" (UniqueName: \"kubernetes.io/projected/39f7d8ee-53f8-490e-b411-682873f59856-kube-api-access-jdcq4\") pod \"maas-api-revoke-sub-d3fba056-h7nfs\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:30.024141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:30.024112 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:30.408271 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:30.408240 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs"] Nov 02 22:13:30.408720 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:13:30.408682 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f7d8ee_53f8_490e_b411_682873f59856.slice/crio-ee3584e2dece28a76582695dffc49159f9d5d6a6db9918600285474fd21e1ac3 WatchSource:0}: Error finding container ee3584e2dece28a76582695dffc49159f9d5d6a6db9918600285474fd21e1ac3: Status 404 returned error can't find the container with id ee3584e2dece28a76582695dffc49159f9d5d6a6db9918600285474fd21e1ac3 Nov 02 22:13:30.698263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:30.698218 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" event={"ID":"39f7d8ee-53f8-490e-b411-682873f59856","Type":"ContainerStarted","Data":"ee3584e2dece28a76582695dffc49159f9d5d6a6db9918600285474fd21e1ac3"} Nov 02 22:13:30.914238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:30.914208 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 22:13:30.914629 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:30.914257 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 22:13:31.713065 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:31.708917 4441 generic.go:356] "Generic (PLEG): container finished" podID="39f7d8ee-53f8-490e-b411-682873f59856" containerID="dcca7e1b01c74a003f65a597d9ad951a5fb11733b657bf5fd92a98ca60a5af58" exitCode=0 Nov 02 22:13:31.713065 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:31.709059 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" event={"ID":"39f7d8ee-53f8-490e-b411-682873f59856","Type":"ContainerDied","Data":"dcca7e1b01c74a003f65a597d9ad951a5fb11733b657bf5fd92a98ca60a5af58"} Nov 02 22:13:31.913787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:31.913750 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 22:13:31.914017 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:31.913811 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 22:13:32.914779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:32.914318 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 22:13:32.914779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:32.914379 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 22:13:32.940981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:32.940943 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:33.046270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.046172 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f7d8ee-53f8-490e-b411-682873f59856-kube-api-access-jdcq4\" (UniqueName: \"kubernetes.io/projected/39f7d8ee-53f8-490e-b411-682873f59856-kube-api-access-jdcq4\") pod \"39f7d8ee-53f8-490e-b411-682873f59856\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " Nov 02 22:13:33.046270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.046244 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-openshift-ca\") pod \"39f7d8ee-53f8-490e-b411-682873f59856\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " Nov 02 22:13:33.046596 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.046297 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-xks-ca\") pod \"39f7d8ee-53f8-490e-b411-682873f59856\" (UID: \"39f7d8ee-53f8-490e-b411-682873f59856\") " Nov 02 22:13:33.046659 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.046623 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-openshift-ca" pod "39f7d8ee-53f8-490e-b411-682873f59856" (UID: "39f7d8ee-53f8-490e-b411-682873f59856"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:33.046806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.046787 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.048743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.048706 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-xks-ca" pod "39f7d8ee-53f8-490e-b411-682873f59856" (UID: "39f7d8ee-53f8-490e-b411-682873f59856"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:33.048953 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.048935 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f7d8ee-53f8-490e-b411-682873f59856-kube-api-access-jdcq4" pod "39f7d8ee-53f8-490e-b411-682873f59856" (UID: "39f7d8ee-53f8-490e-b411-682873f59856"). InnerVolumeSpecName "kube-api-access-jdcq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:33.147587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.147548 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcq4\" (UniqueName: \"kubernetes.io/projected/39f7d8ee-53f8-490e-b411-682873f59856-kube-api-access-jdcq4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.147587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.147578 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f7d8ee-53f8-490e-b411-682873f59856-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.738994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.735682 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" event={"ID":"39f7d8ee-53f8-490e-b411-682873f59856","Type":"ContainerDied","Data":"ee3584e2dece28a76582695dffc49159f9d5d6a6db9918600285474fd21e1ac3"} Nov 02 22:13:33.738994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.735748 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3584e2dece28a76582695dffc49159f9d5d6a6db9918600285474fd21e1ac3" Nov 02 22:13:33.738994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.735820 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs" Nov 02 22:13:33.924315 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.924276 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 22:13:33.924961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:33.924919 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 22:13:34.914991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:34.914300 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 22:13:34.914991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:34.914366 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 22:13:35.914103 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:35.914067 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 22:13:35.914529 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:35.914119 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 22:13:37.914076 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:37.914040 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:37.914536 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:37.914091 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.955125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:38.955086 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:38.955612 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:38.955158 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:39.192839 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:39.192774 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:39.193275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:39.193252 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:39.194469 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:13:39.194447 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" Nov 02 22:13:40.614415 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:13:40.614376 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:14:01.062868 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:01.062810 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:04.353420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.353386 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g"] Nov 02 22:14:04.353829 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.353787 4441 cpu_manager.go:414] "containerMap: removing container" podUID="39f7d8ee-53f8-490e-b411-682873f59856" containerName="revoke-keys" Nov 02 22:14:04.353829 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.353799 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f7d8ee-53f8-490e-b411-682873f59856" containerName="revoke-keys" Nov 02 22:14:04.353923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.353877 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f7d8ee-53f8-490e-b411-682873f59856" containerName="revoke-keys" Nov 02 22:14:04.357080 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.357061 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.411800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.411754 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g"] Nov 02 22:14:04.430184 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.430155 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.430343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.430206 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wml82\" (UniqueName: \"kubernetes.io/projected/6f5af569-a2c4-453a-b261-6ca0033b8da1-kube-api-access-wml82\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.430387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.430365 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.453491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.453455 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:04.453491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.453490 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:04.453739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.453558 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:04.454115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.454088 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:14:04.531376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.531339 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.531567 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.531391 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.531567 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.531427 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wml82\" (UniqueName: \"kubernetes.io/projected/6f5af569-a2c4-453a-b261-6ca0033b8da1-kube-api-access-wml82\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.532095 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.532073 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.533848 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.533821 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.592071 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.592040 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wml82\" (UniqueName: \"kubernetes.io/projected/6f5af569-a2c4-453a-b261-6ca0033b8da1-kube-api-access-wml82\") pod \"maas-api-revoke-sub-ae594979-gcr2g\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:04.670420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:04.670340 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:05.038434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:05.038403 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g"] Nov 02 22:14:05.039189 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:14:05.039155 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5af569_a2c4_453a_b261_6ca0033b8da1.slice/crio-a2b62e2c52ec48ffe2716cea2e191e7db38e2c6a3e95480fafd7dfeae119b90b WatchSource:0}: Error finding container a2b62e2c52ec48ffe2716cea2e191e7db38e2c6a3e95480fafd7dfeae119b90b: Status 404 returned error can't find the container with id a2b62e2c52ec48ffe2716cea2e191e7db38e2c6a3e95480fafd7dfeae119b90b Nov 02 22:14:05.960490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:05.960459 4441 generic.go:356] "Generic (PLEG): container finished" podID="6f5af569-a2c4-453a-b261-6ca0033b8da1" containerID="0a8110a2ef3a3ba089799499d967a2cf3885a965862398c9debddcde0464d12d" exitCode=0 Nov 02 22:14:05.960831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:05.960546 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" event={"ID":"6f5af569-a2c4-453a-b261-6ca0033b8da1","Type":"ContainerDied","Data":"0a8110a2ef3a3ba089799499d967a2cf3885a965862398c9debddcde0464d12d"} Nov 02 22:14:05.960831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:05.960609 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" event={"ID":"6f5af569-a2c4-453a-b261-6ca0033b8da1","Type":"ContainerStarted","Data":"a2b62e2c52ec48ffe2716cea2e191e7db38e2c6a3e95480fafd7dfeae119b90b"} Nov 02 22:14:07.097233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.097210 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:07.156342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.156318 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-xks-ca\") pod \"6f5af569-a2c4-453a-b261-6ca0033b8da1\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " Nov 02 22:14:07.156480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.156361 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5af569-a2c4-453a-b261-6ca0033b8da1-kube-api-access-wml82\" (UniqueName: \"kubernetes.io/projected/6f5af569-a2c4-453a-b261-6ca0033b8da1-kube-api-access-wml82\") pod \"6f5af569-a2c4-453a-b261-6ca0033b8da1\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " Nov 02 22:14:07.156480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.156394 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-openshift-ca\") pod \"6f5af569-a2c4-453a-b261-6ca0033b8da1\" (UID: \"6f5af569-a2c4-453a-b261-6ca0033b8da1\") " Nov 02 22:14:07.156798 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.156776 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-openshift-ca" pod "6f5af569-a2c4-453a-b261-6ca0033b8da1" (UID: "6f5af569-a2c4-453a-b261-6ca0033b8da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:07.158299 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.158273 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-xks-ca" pod "6f5af569-a2c4-453a-b261-6ca0033b8da1" (UID: "6f5af569-a2c4-453a-b261-6ca0033b8da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:07.158421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.158406 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5af569-a2c4-453a-b261-6ca0033b8da1-kube-api-access-wml82" pod "6f5af569-a2c4-453a-b261-6ca0033b8da1" (UID: "6f5af569-a2c4-453a-b261-6ca0033b8da1"). InnerVolumeSpecName "kube-api-access-wml82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:07.258092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.258027 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:07.258092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.258055 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wml82\" (UniqueName: \"kubernetes.io/projected/6f5af569-a2c4-453a-b261-6ca0033b8da1-kube-api-access-wml82\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:07.258092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.258066 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5af569-a2c4-453a-b261-6ca0033b8da1-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:07.972707 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.972676 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" Nov 02 22:14:07.972890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.972721 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g" event={"ID":"6f5af569-a2c4-453a-b261-6ca0033b8da1","Type":"ContainerDied","Data":"a2b62e2c52ec48ffe2716cea2e191e7db38e2c6a3e95480fafd7dfeae119b90b"} Nov 02 22:14:07.972890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:07.972768 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2b62e2c52ec48ffe2716cea2e191e7db38e2c6a3e95480fafd7dfeae119b90b" Nov 02 22:14:10.614676 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:14:10.614643 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:14:19.549020 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:19.548955 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:22.616168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:22.616125 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:22.616828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:22.616783 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:28.951198 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:28.951161 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw"] Nov 02 22:14:28.951748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:28.951726 4441 cpu_manager.go:414] "containerMap: removing container" podUID="6f5af569-a2c4-453a-b261-6ca0033b8da1" containerName="revoke-keys" Nov 02 22:14:28.951828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:28.951751 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5af569-a2c4-453a-b261-6ca0033b8da1" containerName="revoke-keys" Nov 02 22:14:28.951898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:28.951887 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5af569-a2c4-453a-b261-6ca0033b8da1" containerName="revoke-keys" Nov 02 22:14:28.955179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:28.955149 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.008846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.008813 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw"] Nov 02 22:14:29.052015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.051984 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:29.052015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.052003 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:14:29.052015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.051996 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:29.052325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.051992 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:29.113304 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.113279 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.113447 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.113324 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gzw\" (UniqueName: \"kubernetes.io/projected/770342b7-c7f6-4beb-a429-4e41d9610fdf-kube-api-access-c8gzw\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.113447 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.113346 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.214046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.213934 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.214046 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.214025 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gzw\" (UniqueName: \"kubernetes.io/projected/770342b7-c7f6-4beb-a429-4e41d9610fdf-kube-api-access-c8gzw\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.214315 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.214053 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.214767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.214740 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.216350 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.216328 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.268018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.267984 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gzw\" (UniqueName: \"kubernetes.io/projected/770342b7-c7f6-4beb-a429-4e41d9610fdf-kube-api-access-c8gzw\") pod \"maas-api-revoke-sub-a4aef7d7-mt4rw\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.268150 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.268114 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:29.638182 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:29.638155 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw"] Nov 02 22:14:29.639127 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:14:29.639097 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770342b7_c7f6_4beb_a429_4e41d9610fdf.slice/crio-9ab2aa2ce083ba251267e461511dcc0c527ef2d01a4d3b3b4e3913974267eb84 WatchSource:0}: Error finding container 9ab2aa2ce083ba251267e461511dcc0c527ef2d01a4d3b3b4e3913974267eb84: Status 404 returned error can't find the container with id 9ab2aa2ce083ba251267e461511dcc0c527ef2d01a4d3b3b4e3913974267eb84 Nov 02 22:14:30.126827 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:30.126790 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" event={"ID":"770342b7-c7f6-4beb-a429-4e41d9610fdf","Type":"ContainerStarted","Data":"37ade576fddec70ec8063ed360fb6c1c8486debb4976c48504afeb315c02bb0f"} Nov 02 22:14:30.127298 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:30.126844 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" event={"ID":"770342b7-c7f6-4beb-a429-4e41d9610fdf","Type":"ContainerStarted","Data":"9ab2aa2ce083ba251267e461511dcc0c527ef2d01a4d3b3b4e3913974267eb84"} Nov 02 22:14:30.236624 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:30.236569 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" podStartSLOduration=2.23655462 podStartE2EDuration="2.23655462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:30.234073268 +0000 UTC m=+3699.789701603" watchObservedRunningTime="2026-11-02 22:14:30.23655462 +0000 UTC m=+3699.792182957" Nov 02 22:14:31.135197 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:31.135162 4441 generic.go:356] "Generic (PLEG): container finished" podID="770342b7-c7f6-4beb-a429-4e41d9610fdf" containerID="37ade576fddec70ec8063ed360fb6c1c8486debb4976c48504afeb315c02bb0f" exitCode=0 Nov 02 22:14:31.135619 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:31.135242 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" event={"ID":"770342b7-c7f6-4beb-a429-4e41d9610fdf","Type":"ContainerDied","Data":"37ade576fddec70ec8063ed360fb6c1c8486debb4976c48504afeb315c02bb0f"} Nov 02 22:14:32.278734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.278711 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:32.340957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.340908 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770342b7-c7f6-4beb-a429-4e41d9610fdf-kube-api-access-c8gzw\" (UniqueName: \"kubernetes.io/projected/770342b7-c7f6-4beb-a429-4e41d9610fdf-kube-api-access-c8gzw\") pod \"770342b7-c7f6-4beb-a429-4e41d9610fdf\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " Nov 02 22:14:32.341096 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.340998 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-openshift-ca\") pod \"770342b7-c7f6-4beb-a429-4e41d9610fdf\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " Nov 02 22:14:32.341096 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.341073 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-xks-ca\") pod \"770342b7-c7f6-4beb-a429-4e41d9610fdf\" (UID: \"770342b7-c7f6-4beb-a429-4e41d9610fdf\") " Nov 02 22:14:32.341458 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.341366 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-openshift-ca" pod "770342b7-c7f6-4beb-a429-4e41d9610fdf" (UID: "770342b7-c7f6-4beb-a429-4e41d9610fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:32.343260 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.343237 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-xks-ca" pod "770342b7-c7f6-4beb-a429-4e41d9610fdf" (UID: "770342b7-c7f6-4beb-a429-4e41d9610fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:32.343430 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.343402 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770342b7-c7f6-4beb-a429-4e41d9610fdf-kube-api-access-c8gzw" pod "770342b7-c7f6-4beb-a429-4e41d9610fdf" (UID: "770342b7-c7f6-4beb-a429-4e41d9610fdf"). InnerVolumeSpecName "kube-api-access-c8gzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:32.441994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.441890 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:32.441994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.441922 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gzw\" (UniqueName: \"kubernetes.io/projected/770342b7-c7f6-4beb-a429-4e41d9610fdf-kube-api-access-c8gzw\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:32.441994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:32.441937 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770342b7-c7f6-4beb-a429-4e41d9610fdf-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:33.148419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:33.148381 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" event={"ID":"770342b7-c7f6-4beb-a429-4e41d9610fdf","Type":"ContainerDied","Data":"9ab2aa2ce083ba251267e461511dcc0c527ef2d01a4d3b3b4e3913974267eb84"} Nov 02 22:14:33.148648 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:33.148454 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab2aa2ce083ba251267e461511dcc0c527ef2d01a4d3b3b4e3913974267eb84" Nov 02 22:14:33.148648 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:33.148390 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw" Nov 02 22:14:40.614707 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:14:40.614667 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:14:55.586810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:55.586774 4441 scope.go:122] "RemoveContainer" containerID="69649c6d6b19c9e70e5295dd88fa569a94464a53f8878d9a6c604f051eba6ff8" Nov 02 22:14:55.599199 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:55.599148 4441 scope.go:122] "RemoveContainer" containerID="0f7bbb16984989c1a3661deee3a365db20447bed9c43a7b7c7dd7dff15bc59fc" Nov 02 22:14:55.607944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:14:55.607923 4441 scope.go:122] "RemoveContainer" containerID="62706814850bfe9771481a91b5e300e3b26a5419ae92479858b36a1192ab6061" Nov 02 22:15:00.153235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.153192 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc"] Nov 02 22:15:00.153761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.153667 4441 cpu_manager.go:414] "containerMap: removing container" podUID="770342b7-c7f6-4beb-a429-4e41d9610fdf" containerName="revoke-keys" Nov 02 22:15:00.153761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.153679 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770342b7-c7f6-4beb-a429-4e41d9610fdf" containerName="revoke-keys" Nov 02 22:15:00.153761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.153758 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="770342b7-c7f6-4beb-a429-4e41d9610fdf" containerName="revoke-keys" Nov 02 22:15:00.157058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.157033 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.206833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.206795 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:15:00.207007 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.206872 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:15:00.254175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.254128 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:15:00.254388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.254248 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:15:00.259104 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.259076 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc"] Nov 02 22:15:00.352001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.351948 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7k4\" (UniqueName: \"kubernetes.io/projected/48314b43-3313-42d0-8681-034df736a477-kube-api-access-pq7k4\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.352144 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.352012 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48314b43-3313-42d0-8681-034df736a477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.352144 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.352090 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48314b43-3313-42d0-8681-034df736a477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.452997 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.452918 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7k4\" (UniqueName: \"kubernetes.io/projected/48314b43-3313-42d0-8681-034df736a477-kube-api-access-pq7k4\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.452997 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.452952 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48314b43-3313-42d0-8681-034df736a477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.453184 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.453016 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48314b43-3313-42d0-8681-034df736a477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.453644 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.453621 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48314b43-3313-42d0-8681-034df736a477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.455363 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.455342 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48314b43-3313-42d0-8681-034df736a477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.506627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.506604 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7k4\" (UniqueName: \"kubernetes.io/projected/48314b43-3313-42d0-8681-034df736a477-kube-api-access-pq7k4\") pod \"maas-api-revoke-sub-79805ca9-gnfwc\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:00.770170 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:00.770080 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:01.135080 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:01.135047 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc"] Nov 02 22:15:01.136988 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:15:01.136934 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48314b43_3313_42d0_8681_034df736a477.slice/crio-8c0a383b795323d39738d046eda021e5421035de0409cf9a9c244a27d93df571 WatchSource:0}: Error finding container 8c0a383b795323d39738d046eda021e5421035de0409cf9a9c244a27d93df571: Status 404 returned error can't find the container with id 8c0a383b795323d39738d046eda021e5421035de0409cf9a9c244a27d93df571 Nov 02 22:15:01.364378 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:01.364343 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" event={"ID":"48314b43-3313-42d0-8681-034df736a477","Type":"ContainerStarted","Data":"8c0a383b795323d39738d046eda021e5421035de0409cf9a9c244a27d93df571"} Nov 02 22:15:01.368539 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:01.368510 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:02.371328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:02.371292 4441 generic.go:356] "Generic (PLEG): container finished" podID="48314b43-3313-42d0-8681-034df736a477" containerID="a4b296f64e4258cb810696eb2d5ca23e81edcaad3466228471c736b704b985e0" exitCode=0 Nov 02 22:15:02.371751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:02.371362 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" event={"ID":"48314b43-3313-42d0-8681-034df736a477","Type":"ContainerDied","Data":"a4b296f64e4258cb810696eb2d5ca23e81edcaad3466228471c736b704b985e0"} Nov 02 22:15:03.518843 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.518816 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:03.581685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.581655 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48314b43-3313-42d0-8681-034df736a477-kube-api-access-pq7k4\" (UniqueName: \"kubernetes.io/projected/48314b43-3313-42d0-8681-034df736a477-kube-api-access-pq7k4\") pod \"48314b43-3313-42d0-8681-034df736a477\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " Nov 02 22:15:03.581856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.581783 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48314b43-3313-42d0-8681-034df736a477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48314b43-3313-42d0-8681-034df736a477-maas-api-openshift-ca\") pod \"48314b43-3313-42d0-8681-034df736a477\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " Nov 02 22:15:03.581856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.581810 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48314b43-3313-42d0-8681-034df736a477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48314b43-3313-42d0-8681-034df736a477-maas-api-xks-ca\") pod \"48314b43-3313-42d0-8681-034df736a477\" (UID: \"48314b43-3313-42d0-8681-034df736a477\") " Nov 02 22:15:03.582175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.582149 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48314b43-3313-42d0-8681-034df736a477-maas-api-openshift-ca" pod "48314b43-3313-42d0-8681-034df736a477" (UID: "48314b43-3313-42d0-8681-034df736a477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:03.583738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.583714 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48314b43-3313-42d0-8681-034df736a477-kube-api-access-pq7k4" pod "48314b43-3313-42d0-8681-034df736a477" (UID: "48314b43-3313-42d0-8681-034df736a477"). InnerVolumeSpecName "kube-api-access-pq7k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:03.583810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.583773 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48314b43-3313-42d0-8681-034df736a477-maas-api-xks-ca" pod "48314b43-3313-42d0-8681-034df736a477" (UID: "48314b43-3313-42d0-8681-034df736a477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:03.683220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.683147 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48314b43-3313-42d0-8681-034df736a477-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:03.683220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.683177 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48314b43-3313-42d0-8681-034df736a477-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:03.683220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:03.683186 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7k4\" (UniqueName: \"kubernetes.io/projected/48314b43-3313-42d0-8681-034df736a477-kube-api-access-pq7k4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:04.053238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:04.053152 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42"] Nov 02 22:15:04.055583 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:04.055557 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171072d1-gfw42"] Nov 02 22:15:04.384461 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:04.384431 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" Nov 02 22:15:04.384668 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:04.384470 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc" event={"ID":"48314b43-3313-42d0-8681-034df736a477","Type":"ContainerDied","Data":"8c0a383b795323d39738d046eda021e5421035de0409cf9a9c244a27d93df571"} Nov 02 22:15:04.384668 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:04.384525 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0a383b795323d39738d046eda021e5421035de0409cf9a9c244a27d93df571" Nov 02 22:15:04.583133 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:04.583104 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1bf94db-222d-4d4a-ac2b-c0024935d7ca" path="/var/lib/kubelet/pods/b1bf94db-222d-4d4a-ac2b-c0024935d7ca/volumes" Nov 02 22:15:10.615138 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:15:10.615100 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:15:23.828749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.828715 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6"] Nov 02 22:15:23.829167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.829112 4441 cpu_manager.go:414] "containerMap: removing container" podUID="48314b43-3313-42d0-8681-034df736a477" containerName="revoke-keys" Nov 02 22:15:23.829167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.829125 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48314b43-3313-42d0-8681-034df736a477" containerName="revoke-keys" Nov 02 22:15:23.829248 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.829197 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="48314b43-3313-42d0-8681-034df736a477" containerName="revoke-keys" Nov 02 22:15:23.833630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.833611 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:23.885750 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.885727 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6"] Nov 02 22:15:23.903066 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.903031 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:23.903208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.903124 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmlv\" (UniqueName: \"kubernetes.io/projected/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-kube-api-access-rqmlv\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:23.903208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.903177 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:23.929366 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.929343 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:15:23.929366 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.929345 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:15:23.929524 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.929388 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:15:23.930452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:23.930437 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:15:24.003602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.003571 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:24.003758 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.003624 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmlv\" (UniqueName: \"kubernetes.io/projected/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-kube-api-access-rqmlv\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:24.003758 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.003664 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:24.004271 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.004245 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:24.005957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.005934 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:24.058092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.058062 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmlv\" (UniqueName: \"kubernetes.io/projected/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-kube-api-access-rqmlv\") pod \"maas-api-revoke-sub-0b7e8edb-b6dc6\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:24.146566 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.146509 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:24.510061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.510029 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6"] Nov 02 22:15:24.510641 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:15:24.510617 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96666b63_4e41_4bbf_a3b8_3c0350f8de7b.slice/crio-c2665dc5836ef21c6beda14d7e9b472baf9569f8138a7ed14972dbd34cde1e07 WatchSource:0}: Error finding container c2665dc5836ef21c6beda14d7e9b472baf9569f8138a7ed14972dbd34cde1e07: Status 404 returned error can't find the container with id c2665dc5836ef21c6beda14d7e9b472baf9569f8138a7ed14972dbd34cde1e07 Nov 02 22:15:24.532099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:24.532071 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" event={"ID":"96666b63-4e41-4bbf-a3b8-3c0350f8de7b","Type":"ContainerStarted","Data":"c2665dc5836ef21c6beda14d7e9b472baf9569f8138a7ed14972dbd34cde1e07"} Nov 02 22:15:25.539963 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:25.539931 4441 generic.go:356] "Generic (PLEG): container finished" podID="96666b63-4e41-4bbf-a3b8-3c0350f8de7b" containerID="d2d6bb19e0c9a266996b5b65b64f55ffb5afa333bb6a09430ed760e878f94d3a" exitCode=0 Nov 02 22:15:25.540330 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:25.539995 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" event={"ID":"96666b63-4e41-4bbf-a3b8-3c0350f8de7b","Type":"ContainerDied","Data":"d2d6bb19e0c9a266996b5b65b64f55ffb5afa333bb6a09430ed760e878f94d3a"} Nov 02 22:15:26.677723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.677699 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:26.728215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.728191 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-kube-api-access-rqmlv\" (UniqueName: \"kubernetes.io/projected/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-kube-api-access-rqmlv\") pod \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " Nov 02 22:15:26.728332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.728230 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-xks-ca\") pod \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " Nov 02 22:15:26.728332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.728311 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-openshift-ca\") pod \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\" (UID: \"96666b63-4e41-4bbf-a3b8-3c0350f8de7b\") " Nov 02 22:15:26.728695 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.728668 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-openshift-ca" pod "96666b63-4e41-4bbf-a3b8-3c0350f8de7b" (UID: "96666b63-4e41-4bbf-a3b8-3c0350f8de7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:26.730311 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.730283 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-xks-ca" pod "96666b63-4e41-4bbf-a3b8-3c0350f8de7b" (UID: "96666b63-4e41-4bbf-a3b8-3c0350f8de7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:26.730418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.730369 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-kube-api-access-rqmlv" pod "96666b63-4e41-4bbf-a3b8-3c0350f8de7b" (UID: "96666b63-4e41-4bbf-a3b8-3c0350f8de7b"). InnerVolumeSpecName "kube-api-access-rqmlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:26.829890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.829857 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:26.829890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.829886 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:26.830085 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:26.829902 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmlv\" (UniqueName: \"kubernetes.io/projected/96666b63-4e41-4bbf-a3b8-3c0350f8de7b-kube-api-access-rqmlv\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:27.552122 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:27.552091 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" Nov 02 22:15:27.552303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:27.552124 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6" event={"ID":"96666b63-4e41-4bbf-a3b8-3c0350f8de7b","Type":"ContainerDied","Data":"c2665dc5836ef21c6beda14d7e9b472baf9569f8138a7ed14972dbd34cde1e07"} Nov 02 22:15:27.552303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:27.552176 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2665dc5836ef21c6beda14d7e9b472baf9569f8138a7ed14972dbd34cde1e07" Nov 02 22:15:35.198130 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.198095 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7"] Nov 02 22:15:35.198652 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.198465 4441 cpu_manager.go:414] "containerMap: removing container" podUID="96666b63-4e41-4bbf-a3b8-3c0350f8de7b" containerName="revoke-keys" Nov 02 22:15:35.198652 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.198476 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96666b63-4e41-4bbf-a3b8-3c0350f8de7b" containerName="revoke-keys" Nov 02 22:15:35.198652 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.198558 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="96666b63-4e41-4bbf-a3b8-3c0350f8de7b" containerName="revoke-keys" Nov 02 22:15:35.202067 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.202037 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.256580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.256539 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7"] Nov 02 22:15:35.298368 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.298334 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:15:35.298815 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.298800 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:15:35.298881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.298830 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:15:35.298881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.298833 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:15:35.305402 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.305378 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5627m\" (UniqueName: \"kubernetes.io/projected/4856062f-56f5-48ce-b3f7-6c3be090edd6-kube-api-access-5627m\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.305516 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.305493 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.305564 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.305539 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.406273 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.406245 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.406389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.406285 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.406389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.406332 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5627m\" (UniqueName: \"kubernetes.io/projected/4856062f-56f5-48ce-b3f7-6c3be090edd6-kube-api-access-5627m\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.407034 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.407018 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.408499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.408477 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.460092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.460034 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5627m\" (UniqueName: \"kubernetes.io/projected/4856062f-56f5-48ce-b3f7-6c3be090edd6-kube-api-access-5627m\") pod \"maas-api-revoke-sub-3d4156f1-2xzb7\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.515256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.515233 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:35.881808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:35.881772 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7"] Nov 02 22:15:35.882489 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:15:35.882456 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4856062f_56f5_48ce_b3f7_6c3be090edd6.slice/crio-be07793451237ee5f1bfb605b92a41fa8d2ffeea7dcb8d250af208a7f995359d WatchSource:0}: Error finding container be07793451237ee5f1bfb605b92a41fa8d2ffeea7dcb8d250af208a7f995359d: Status 404 returned error can't find the container with id be07793451237ee5f1bfb605b92a41fa8d2ffeea7dcb8d250af208a7f995359d Nov 02 22:15:36.605295 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:36.605260 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" event={"ID":"4856062f-56f5-48ce-b3f7-6c3be090edd6","Type":"ContainerStarted","Data":"b0e6c1ecdfbdb995b9ca90ace5c34f789b6e46640107e677bfd53454fe81002d"} Nov 02 22:15:36.605698 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:36.605316 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" event={"ID":"4856062f-56f5-48ce-b3f7-6c3be090edd6","Type":"ContainerStarted","Data":"be07793451237ee5f1bfb605b92a41fa8d2ffeea7dcb8d250af208a7f995359d"} Nov 02 22:15:36.714202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:36.714147 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" podStartSLOduration=1.714133682 podStartE2EDuration="1.714133682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:36.711534127 +0000 UTC m=+3766.267162461" watchObservedRunningTime="2026-11-02 22:15:36.714133682 +0000 UTC m=+3766.269762020" Nov 02 22:15:37.630992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:37.630691 4441 generic.go:356] "Generic (PLEG): container finished" podID="4856062f-56f5-48ce-b3f7-6c3be090edd6" containerID="b0e6c1ecdfbdb995b9ca90ace5c34f789b6e46640107e677bfd53454fe81002d" exitCode=0 Nov 02 22:15:37.630992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:37.630750 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" event={"ID":"4856062f-56f5-48ce-b3f7-6c3be090edd6","Type":"ContainerDied","Data":"b0e6c1ecdfbdb995b9ca90ace5c34f789b6e46640107e677bfd53454fe81002d"} Nov 02 22:15:38.749207 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:38.749158 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:15:38.749724 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:38.749242 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:15:39.324173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.324102 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:39.400938 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.400901 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-xks-ca\") pod \"4856062f-56f5-48ce-b3f7-6c3be090edd6\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " Nov 02 22:15:39.401173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.400949 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-openshift-ca\") pod \"4856062f-56f5-48ce-b3f7-6c3be090edd6\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " Nov 02 22:15:39.401173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.401068 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4856062f-56f5-48ce-b3f7-6c3be090edd6-kube-api-access-5627m\" (UniqueName: \"kubernetes.io/projected/4856062f-56f5-48ce-b3f7-6c3be090edd6-kube-api-access-5627m\") pod \"4856062f-56f5-48ce-b3f7-6c3be090edd6\" (UID: \"4856062f-56f5-48ce-b3f7-6c3be090edd6\") " Nov 02 22:15:39.401400 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.401370 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-openshift-ca" pod "4856062f-56f5-48ce-b3f7-6c3be090edd6" (UID: "4856062f-56f5-48ce-b3f7-6c3be090edd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:39.403106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.403076 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-xks-ca" pod "4856062f-56f5-48ce-b3f7-6c3be090edd6" (UID: "4856062f-56f5-48ce-b3f7-6c3be090edd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:39.403571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.403551 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4856062f-56f5-48ce-b3f7-6c3be090edd6-kube-api-access-5627m" pod "4856062f-56f5-48ce-b3f7-6c3be090edd6" (UID: "4856062f-56f5-48ce-b3f7-6c3be090edd6"). InnerVolumeSpecName "kube-api-access-5627m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:39.502477 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.502441 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5627m\" (UniqueName: \"kubernetes.io/projected/4856062f-56f5-48ce-b3f7-6c3be090edd6-kube-api-access-5627m\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:39.502477 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.502473 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:39.502746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.502487 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4856062f-56f5-48ce-b3f7-6c3be090edd6-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:39.670483 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.670456 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" Nov 02 22:15:39.670654 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.670495 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7" event={"ID":"4856062f-56f5-48ce-b3f7-6c3be090edd6","Type":"ContainerDied","Data":"be07793451237ee5f1bfb605b92a41fa8d2ffeea7dcb8d250af208a7f995359d"} Nov 02 22:15:39.670654 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:39.670540 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be07793451237ee5f1bfb605b92a41fa8d2ffeea7dcb8d250af208a7f995359d" Nov 02 22:15:40.615099 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:15:40.615056 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:15:53.054693 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.054659 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7"] Nov 02 22:15:53.101906 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.101876 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2xzb7"] Nov 02 22:15:53.167161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.167127 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd"] Nov 02 22:15:53.167722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.167703 4441 cpu_manager.go:414] "containerMap: removing container" podUID="4856062f-56f5-48ce-b3f7-6c3be090edd6" containerName="revoke-keys" Nov 02 22:15:53.167810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.167723 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4856062f-56f5-48ce-b3f7-6c3be090edd6" containerName="revoke-keys" Nov 02 22:15:53.167854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.167841 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="4856062f-56f5-48ce-b3f7-6c3be090edd6" containerName="revoke-keys" Nov 02 22:15:53.171685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.171664 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.220650 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.220613 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd"] Nov 02 22:15:53.221223 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.221198 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:15:53.221223 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.221205 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:15:53.221418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.221287 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:15:53.221418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.221369 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:15:53.230729 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.230697 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.230816 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.230744 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6gm4\" (UniqueName: \"kubernetes.io/projected/ee529a0d-a0bd-4552-8009-12c893200863-kube-api-access-b6gm4\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.230816 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.230790 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.331666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.331620 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.331830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.331692 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6gm4\" (UniqueName: \"kubernetes.io/projected/ee529a0d-a0bd-4552-8009-12c893200863-kube-api-access-b6gm4\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.331830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.331718 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.332395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.332372 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.334001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.333979 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.384989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.384957 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6gm4\" (UniqueName: \"kubernetes.io/projected/ee529a0d-a0bd-4552-8009-12c893200863-kube-api-access-b6gm4\") pod \"maas-api-revoke-sub-3d4156f1-4c2nd\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.485204 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.485179 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:53.848382 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:15:53.848354 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee529a0d_a0bd_4552_8009_12c893200863.slice/crio-7c41ae804e431085bbf628104f31df39c8ee50039f7fca5484f6d5a5e20da469 WatchSource:0}: Error finding container 7c41ae804e431085bbf628104f31df39c8ee50039f7fca5484f6d5a5e20da469: Status 404 returned error can't find the container with id 7c41ae804e431085bbf628104f31df39c8ee50039f7fca5484f6d5a5e20da469 Nov 02 22:15:53.848757 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:53.848728 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd"] Nov 02 22:15:54.582810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:54.582776 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4856062f-56f5-48ce-b3f7-6c3be090edd6" path="/var/lib/kubelet/pods/4856062f-56f5-48ce-b3f7-6c3be090edd6/volumes" Nov 02 22:15:54.756586 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:54.756546 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" event={"ID":"ee529a0d-a0bd-4552-8009-12c893200863","Type":"ContainerStarted","Data":"f1c68e32778e02f82680ad05559b605e58d032e06af9f7c4c839f1baff29cdf0"} Nov 02 22:15:54.756763 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:54.756604 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" event={"ID":"ee529a0d-a0bd-4552-8009-12c893200863","Type":"ContainerStarted","Data":"7c41ae804e431085bbf628104f31df39c8ee50039f7fca5484f6d5a5e20da469"} Nov 02 22:15:54.864712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:54.864622 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" podStartSLOduration=1.8646085860000001 podStartE2EDuration="1.864608586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:54.862694876 +0000 UTC m=+3784.418323210" watchObservedRunningTime="2026-11-02 22:15:54.864608586 +0000 UTC m=+3784.420236927" Nov 02 22:15:55.638044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:55.638012 4441 scope.go:122] "RemoveContainer" containerID="a8998f84173c0da551e74d8cb52d54f27095dacb647fea7a972e4afa65db2617" Nov 02 22:15:55.763463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:55.763429 4441 generic.go:356] "Generic (PLEG): container finished" podID="ee529a0d-a0bd-4552-8009-12c893200863" containerID="f1c68e32778e02f82680ad05559b605e58d032e06af9f7c4c839f1baff29cdf0" exitCode=0 Nov 02 22:15:55.763597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:55.763469 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" event={"ID":"ee529a0d-a0bd-4552-8009-12c893200863","Type":"ContainerDied","Data":"f1c68e32778e02f82680ad05559b605e58d032e06af9f7c4c839f1baff29cdf0"} Nov 02 22:15:56.907957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:56.907935 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:56.964413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:56.964389 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee529a0d-a0bd-4552-8009-12c893200863-kube-api-access-b6gm4\" (UniqueName: \"kubernetes.io/projected/ee529a0d-a0bd-4552-8009-12c893200863-kube-api-access-b6gm4\") pod \"ee529a0d-a0bd-4552-8009-12c893200863\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " Nov 02 22:15:56.964597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:56.964452 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-xks-ca\") pod \"ee529a0d-a0bd-4552-8009-12c893200863\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " Nov 02 22:15:56.964597 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:56.964488 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-openshift-ca\") pod \"ee529a0d-a0bd-4552-8009-12c893200863\" (UID: \"ee529a0d-a0bd-4552-8009-12c893200863\") " Nov 02 22:15:56.964884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:56.964862 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-openshift-ca" pod "ee529a0d-a0bd-4552-8009-12c893200863" (UID: "ee529a0d-a0bd-4552-8009-12c893200863"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:56.966394 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:56.966373 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-xks-ca" pod "ee529a0d-a0bd-4552-8009-12c893200863" (UID: "ee529a0d-a0bd-4552-8009-12c893200863"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:56.966476 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:56.966439 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee529a0d-a0bd-4552-8009-12c893200863-kube-api-access-b6gm4" pod "ee529a0d-a0bd-4552-8009-12c893200863" (UID: "ee529a0d-a0bd-4552-8009-12c893200863"). InnerVolumeSpecName "kube-api-access-b6gm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:57.065776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:57.065746 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6gm4\" (UniqueName: \"kubernetes.io/projected/ee529a0d-a0bd-4552-8009-12c893200863-kube-api-access-b6gm4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:57.065776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:57.065775 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:57.065976 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:57.065790 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee529a0d-a0bd-4552-8009-12c893200863-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:57.775488 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:57.775456 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" Nov 02 22:15:57.775667 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:57.775485 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd" event={"ID":"ee529a0d-a0bd-4552-8009-12c893200863","Type":"ContainerDied","Data":"7c41ae804e431085bbf628104f31df39c8ee50039f7fca5484f6d5a5e20da469"} Nov 02 22:15:57.775667 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:57.775529 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c41ae804e431085bbf628104f31df39c8ee50039f7fca5484f6d5a5e20da469" Nov 02 22:15:59.043495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:59.043461 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc"] Nov 02 22:15:59.090738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:15:59.090709 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjncc"] Nov 02 22:16:00.586673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:00.586639 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d5a06a-2b7a-4fe9-b90a-531b82a970aa" path="/var/lib/kubelet/pods/16d5a06a-2b7a-4fe9-b90a-531b82a970aa/volumes" Nov 02 22:16:01.751135 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:01.751100 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:10.614110 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:16:10.614073 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:16:23.743110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:23.743066 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:16:23.743768 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:23.743735 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zt7w9" podUID="b918a37d-859a-4630-a865-e92121ba6073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:30.629833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.629792 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv"] Nov 02 22:16:30.630409 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.630388 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ee529a0d-a0bd-4552-8009-12c893200863" containerName="revoke-keys" Nov 02 22:16:30.630499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.630412 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee529a0d-a0bd-4552-8009-12c893200863" containerName="revoke-keys" Nov 02 22:16:30.630559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.630543 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee529a0d-a0bd-4552-8009-12c893200863" containerName="revoke-keys" Nov 02 22:16:30.633518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.633497 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.686914 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.686889 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv"] Nov 02 22:16:30.716112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.716081 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qkh\" (UniqueName: \"kubernetes.io/projected/a0dcdff7-ad05-41f1-908a-3a5556aea118-kube-api-access-r8qkh\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.716239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.716220 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.716309 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.716294 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.730445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.730418 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:16:30.730559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.730454 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:16:30.730559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.730478 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:16:30.730559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.730519 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:16:30.816887 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.816866 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.817042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.816916 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.817042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.817027 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qkh\" (UniqueName: \"kubernetes.io/projected/a0dcdff7-ad05-41f1-908a-3a5556aea118-kube-api-access-r8qkh\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.817653 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.817633 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.819275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.819242 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.871375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.871348 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qkh\" (UniqueName: \"kubernetes.io/projected/a0dcdff7-ad05-41f1-908a-3a5556aea118-kube-api-access-r8qkh\") pod \"maas-api-revoke-sub-e2fc9c5b-bqwtv\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:30.945735 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:30.945668 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:31.307658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:31.307619 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv"] Nov 02 22:16:31.308185 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:16:31.308160 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dcdff7_ad05_41f1_908a_3a5556aea118.slice/crio-81aa156e79494d70ce5538a4cf3b63301e71eb3dcec81e1be6fab7b2c65a85cf WatchSource:0}: Error finding container 81aa156e79494d70ce5538a4cf3b63301e71eb3dcec81e1be6fab7b2c65a85cf: Status 404 returned error can't find the container with id 81aa156e79494d70ce5538a4cf3b63301e71eb3dcec81e1be6fab7b2c65a85cf Nov 02 22:16:32.019897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:32.019778 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" event={"ID":"a0dcdff7-ad05-41f1-908a-3a5556aea118","Type":"ContainerStarted","Data":"d08706fafcf9366cf2d4c68ee4208886565934b93ee3e90c893d72d0bb33a42e"} Nov 02 22:16:32.019897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:32.019834 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" event={"ID":"a0dcdff7-ad05-41f1-908a-3a5556aea118","Type":"ContainerStarted","Data":"81aa156e79494d70ce5538a4cf3b63301e71eb3dcec81e1be6fab7b2c65a85cf"} Nov 02 22:16:32.127144 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:32.127092 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" podStartSLOduration=2.127077023 podStartE2EDuration="2.127077023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:32.126345002 +0000 UTC m=+3821.681973336" watchObservedRunningTime="2026-11-02 22:16:32.127077023 +0000 UTC m=+3821.682705360" Nov 02 22:16:33.027286 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:33.027247 4441 generic.go:356] "Generic (PLEG): container finished" podID="a0dcdff7-ad05-41f1-908a-3a5556aea118" containerID="d08706fafcf9366cf2d4c68ee4208886565934b93ee3e90c893d72d0bb33a42e" exitCode=0 Nov 02 22:16:33.027695 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:33.027333 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" event={"ID":"a0dcdff7-ad05-41f1-908a-3a5556aea118","Type":"ContainerDied","Data":"d08706fafcf9366cf2d4c68ee4208886565934b93ee3e90c893d72d0bb33a42e"} Nov 02 22:16:34.154594 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.154563 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:34.245067 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.245032 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-openshift-ca\") pod \"a0dcdff7-ad05-41f1-908a-3a5556aea118\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " Nov 02 22:16:34.245233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.245146 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-xks-ca\") pod \"a0dcdff7-ad05-41f1-908a-3a5556aea118\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " Nov 02 22:16:34.245334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.245318 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dcdff7-ad05-41f1-908a-3a5556aea118-kube-api-access-r8qkh\" (UniqueName: \"kubernetes.io/projected/a0dcdff7-ad05-41f1-908a-3a5556aea118-kube-api-access-r8qkh\") pod \"a0dcdff7-ad05-41f1-908a-3a5556aea118\" (UID: \"a0dcdff7-ad05-41f1-908a-3a5556aea118\") " Nov 02 22:16:34.245404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.245368 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-openshift-ca" pod "a0dcdff7-ad05-41f1-908a-3a5556aea118" (UID: "a0dcdff7-ad05-41f1-908a-3a5556aea118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:34.245673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.245654 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:34.247277 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.247254 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-xks-ca" pod "a0dcdff7-ad05-41f1-908a-3a5556aea118" (UID: "a0dcdff7-ad05-41f1-908a-3a5556aea118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:34.247391 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.247337 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dcdff7-ad05-41f1-908a-3a5556aea118-kube-api-access-r8qkh" pod "a0dcdff7-ad05-41f1-908a-3a5556aea118" (UID: "a0dcdff7-ad05-41f1-908a-3a5556aea118"). InnerVolumeSpecName "kube-api-access-r8qkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:34.347105 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.347076 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0dcdff7-ad05-41f1-908a-3a5556aea118-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:34.347105 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:34.347102 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8qkh\" (UniqueName: \"kubernetes.io/projected/a0dcdff7-ad05-41f1-908a-3a5556aea118-kube-api-access-r8qkh\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:35.040343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:35.040311 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" Nov 02 22:16:35.040343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:35.040339 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv" event={"ID":"a0dcdff7-ad05-41f1-908a-3a5556aea118","Type":"ContainerDied","Data":"81aa156e79494d70ce5538a4cf3b63301e71eb3dcec81e1be6fab7b2c65a85cf"} Nov 02 22:16:35.040645 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:35.040408 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81aa156e79494d70ce5538a4cf3b63301e71eb3dcec81e1be6fab7b2c65a85cf" Nov 02 22:16:40.615276 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:16:40.615233 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:16:41.337025 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.336991 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr"] Nov 02 22:16:41.337414 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.337395 4441 cpu_manager.go:414] "containerMap: removing container" podUID="a0dcdff7-ad05-41f1-908a-3a5556aea118" containerName="revoke-keys" Nov 02 22:16:41.337414 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.337413 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dcdff7-ad05-41f1-908a-3a5556aea118" containerName="revoke-keys" Nov 02 22:16:41.337590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.337495 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dcdff7-ad05-41f1-908a-3a5556aea118" containerName="revoke-keys" Nov 02 22:16:41.340331 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.340307 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.394193 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.394165 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr"] Nov 02 22:16:41.417392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.417362 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvpc\" (UniqueName: \"kubernetes.io/projected/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-kube-api-access-5xvpc\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.417566 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.417408 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.417656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.417554 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.437002 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.436951 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:16:41.437143 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.437030 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:16:41.437143 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.437034 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:16:41.437143 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.437083 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:16:41.518937 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.518910 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.519089 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.518993 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.519089 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.519069 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvpc\" (UniqueName: \"kubernetes.io/projected/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-kube-api-access-5xvpc\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.519533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.519513 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.521408 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.521387 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.572791 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.572762 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvpc\" (UniqueName: \"kubernetes.io/projected/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-kube-api-access-5xvpc\") pod \"maas-api-revoke-sub-5a29efb3-5gqrr\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:41.653037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:41.652962 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:42.016032 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:42.016002 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr"] Nov 02 22:16:42.016578 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:16:42.016542 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab14f90_1a3a_4736_a8e4_574adc5a0ed5.slice/crio-00c61a80dbaaf66dbd8ecded35b1c196d314a40e6617fb6620a1b3a1100049a2 WatchSource:0}: Error finding container 00c61a80dbaaf66dbd8ecded35b1c196d314a40e6617fb6620a1b3a1100049a2: Status 404 returned error can't find the container with id 00c61a80dbaaf66dbd8ecded35b1c196d314a40e6617fb6620a1b3a1100049a2 Nov 02 22:16:42.081994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:42.081947 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" event={"ID":"aab14f90-1a3a-4736-a8e4-574adc5a0ed5","Type":"ContainerStarted","Data":"00c61a80dbaaf66dbd8ecded35b1c196d314a40e6617fb6620a1b3a1100049a2"} Nov 02 22:16:43.090751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:43.090715 4441 generic.go:356] "Generic (PLEG): container finished" podID="aab14f90-1a3a-4736-a8e4-574adc5a0ed5" containerID="20781a55361d06536d4b342c8b7b3a270939bccb0920430a4d9d7e782442f4a4" exitCode=0 Nov 02 22:16:43.091270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:43.090801 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" event={"ID":"aab14f90-1a3a-4736-a8e4-574adc5a0ed5","Type":"ContainerDied","Data":"20781a55361d06536d4b342c8b7b3a270939bccb0920430a4d9d7e782442f4a4"} Nov 02 22:16:44.296037 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.296008 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:44.451236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.451149 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-openshift-ca\") pod \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " Nov 02 22:16:44.451412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.451283 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-xks-ca\") pod \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " Nov 02 22:16:44.451412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.451390 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-kube-api-access-5xvpc\" (UniqueName: \"kubernetes.io/projected/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-kube-api-access-5xvpc\") pod \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\" (UID: \"aab14f90-1a3a-4736-a8e4-574adc5a0ed5\") " Nov 02 22:16:44.452472 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.452440 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-openshift-ca" pod "aab14f90-1a3a-4736-a8e4-574adc5a0ed5" (UID: "aab14f90-1a3a-4736-a8e4-574adc5a0ed5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:44.455223 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.455193 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-xks-ca" pod "aab14f90-1a3a-4736-a8e4-574adc5a0ed5" (UID: "aab14f90-1a3a-4736-a8e4-574adc5a0ed5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:44.467157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.467083 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-kube-api-access-5xvpc" pod "aab14f90-1a3a-4736-a8e4-574adc5a0ed5" (UID: "aab14f90-1a3a-4736-a8e4-574adc5a0ed5"). InnerVolumeSpecName "kube-api-access-5xvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:44.552986 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.552933 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:44.553161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.552989 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:44.553161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:44.553008 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xvpc\" (UniqueName: \"kubernetes.io/projected/aab14f90-1a3a-4736-a8e4-574adc5a0ed5-kube-api-access-5xvpc\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:45.149634 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:45.149592 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" event={"ID":"aab14f90-1a3a-4736-a8e4-574adc5a0ed5","Type":"ContainerDied","Data":"00c61a80dbaaf66dbd8ecded35b1c196d314a40e6617fb6620a1b3a1100049a2"} Nov 02 22:16:45.149931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:45.149909 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c61a80dbaaf66dbd8ecded35b1c196d314a40e6617fb6620a1b3a1100049a2" Nov 02 22:16:45.150167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:45.150150 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr" Nov 02 22:16:55.667944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:55.667909 4441 scope.go:122] "RemoveContainer" containerID="3ae6295bf567b22ab779f52a0cb924267bf762639aef336ad945062b730045f9" Nov 02 22:16:55.677409 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:55.677388 4441 scope.go:122] "RemoveContainer" containerID="b8d5b5265ac5a6300a8ae323735cf8a85f1ce59b4c034d7a5afe820d0756cc89" Nov 02 22:16:59.905808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:59.905772 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd"] Nov 02 22:16:59.953053 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:16:59.953020 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4c2nd"] Nov 02 22:17:00.019314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.019282 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6"] Nov 02 22:17:00.019712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.019692 4441 cpu_manager.go:414] "containerMap: removing container" podUID="aab14f90-1a3a-4736-a8e4-574adc5a0ed5" containerName="revoke-keys" Nov 02 22:17:00.019760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.019717 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab14f90-1a3a-4736-a8e4-574adc5a0ed5" containerName="revoke-keys" Nov 02 22:17:00.019862 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.019851 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab14f90-1a3a-4736-a8e4-574adc5a0ed5" containerName="revoke-keys" Nov 02 22:17:00.023851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.023825 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.073530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.073500 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:17:00.073743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.073529 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:17:00.073743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.073635 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:17:00.073889 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.073663 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:17:00.074207 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.074181 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6"] Nov 02 22:17:00.122261 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.122226 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.122436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.122278 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbnc\" (UniqueName: \"kubernetes.io/projected/ca155641-0b79-4498-806b-7bc00e8fce36-kube-api-access-rxbnc\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.122436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.122307 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.223150 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.223057 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.223150 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.223104 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbnc\" (UniqueName: \"kubernetes.io/projected/ca155641-0b79-4498-806b-7bc00e8fce36-kube-api-access-rxbnc\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.223150 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.223139 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.223918 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.223898 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.225571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.225538 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.276588 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.276559 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbnc\" (UniqueName: \"kubernetes.io/projected/ca155641-0b79-4498-806b-7bc00e8fce36-kube-api-access-rxbnc\") pod \"maas-api-revoke-sub-3d4156f1-xvpk6\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.337189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.337152 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:00.582839 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.582807 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee529a0d-a0bd-4552-8009-12c893200863" path="/var/lib/kubelet/pods/ee529a0d-a0bd-4552-8009-12c893200863/volumes" Nov 02 22:17:00.700526 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:17:00.700483 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca155641_0b79_4498_806b_7bc00e8fce36.slice/crio-4c4ac115a1325495c81ca8c5c09a4b26d875f20265ab6d8c3e3585e70288302c WatchSource:0}: Error finding container 4c4ac115a1325495c81ca8c5c09a4b26d875f20265ab6d8c3e3585e70288302c: Status 404 returned error can't find the container with id 4c4ac115a1325495c81ca8c5c09a4b26d875f20265ab6d8c3e3585e70288302c Nov 02 22:17:00.700723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:00.700696 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6"] Nov 02 22:17:01.260746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:01.260663 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" event={"ID":"ca155641-0b79-4498-806b-7bc00e8fce36","Type":"ContainerStarted","Data":"0abfdc807647f67873364ee49ccf220b7dc96de4be73f536e1ecb4465aadb39d"} Nov 02 22:17:01.260746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:01.260723 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" event={"ID":"ca155641-0b79-4498-806b-7bc00e8fce36","Type":"ContainerStarted","Data":"4c4ac115a1325495c81ca8c5c09a4b26d875f20265ab6d8c3e3585e70288302c"} Nov 02 22:17:01.368721 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:01.368668 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" podStartSLOduration=2.368653443 podStartE2EDuration="2.368653443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:01.366726803 +0000 UTC m=+3850.922355140" watchObservedRunningTime="2026-11-02 22:17:01.368653443 +0000 UTC m=+3850.924281780" Nov 02 22:17:02.012177 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:02.012144 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:02.269575 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:02.269484 4441 generic.go:356] "Generic (PLEG): container finished" podID="ca155641-0b79-4498-806b-7bc00e8fce36" containerID="0abfdc807647f67873364ee49ccf220b7dc96de4be73f536e1ecb4465aadb39d" exitCode=0 Nov 02 22:17:02.269994 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:02.269638 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" event={"ID":"ca155641-0b79-4498-806b-7bc00e8fce36","Type":"ContainerDied","Data":"0abfdc807647f67873364ee49ccf220b7dc96de4be73f536e1ecb4465aadb39d"} Nov 02 22:17:03.408936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.408906 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:03.557332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.557307 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca155641-0b79-4498-806b-7bc00e8fce36-kube-api-access-rxbnc\" (UniqueName: \"kubernetes.io/projected/ca155641-0b79-4498-806b-7bc00e8fce36-kube-api-access-rxbnc\") pod \"ca155641-0b79-4498-806b-7bc00e8fce36\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " Nov 02 22:17:03.557495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.557396 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-xks-ca\") pod \"ca155641-0b79-4498-806b-7bc00e8fce36\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " Nov 02 22:17:03.557495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.557420 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-openshift-ca\") pod \"ca155641-0b79-4498-806b-7bc00e8fce36\" (UID: \"ca155641-0b79-4498-806b-7bc00e8fce36\") " Nov 02 22:17:03.557854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.557829 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-openshift-ca" pod "ca155641-0b79-4498-806b-7bc00e8fce36" (UID: "ca155641-0b79-4498-806b-7bc00e8fce36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:03.559544 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.559519 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-xks-ca" pod "ca155641-0b79-4498-806b-7bc00e8fce36" (UID: "ca155641-0b79-4498-806b-7bc00e8fce36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:03.559612 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.559551 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca155641-0b79-4498-806b-7bc00e8fce36-kube-api-access-rxbnc" pod "ca155641-0b79-4498-806b-7bc00e8fce36" (UID: "ca155641-0b79-4498-806b-7bc00e8fce36"). InnerVolumeSpecName "kube-api-access-rxbnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:03.658648 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.658617 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbnc\" (UniqueName: \"kubernetes.io/projected/ca155641-0b79-4498-806b-7bc00e8fce36-kube-api-access-rxbnc\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:03.658648 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.658644 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:03.658648 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:03.658654 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca155641-0b79-4498-806b-7bc00e8fce36-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:04.282372 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:04.282344 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" Nov 02 22:17:04.282531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:04.282378 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6" event={"ID":"ca155641-0b79-4498-806b-7bc00e8fce36","Type":"ContainerDied","Data":"4c4ac115a1325495c81ca8c5c09a4b26d875f20265ab6d8c3e3585e70288302c"} Nov 02 22:17:04.282531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:04.282425 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4ac115a1325495c81ca8c5c09a4b26d875f20265ab6d8c3e3585e70288302c" Nov 02 22:17:10.614820 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:17:10.614780 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:17:15.888821 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:15.888784 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:16.374399 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.374363 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9"] Nov 02 22:17:16.374822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.374805 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ca155641-0b79-4498-806b-7bc00e8fce36" containerName="revoke-keys" Nov 02 22:17:16.374872 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.374824 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca155641-0b79-4498-806b-7bc00e8fce36" containerName="revoke-keys" Nov 02 22:17:16.374913 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.374883 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca155641-0b79-4498-806b-7bc00e8fce36" containerName="revoke-keys" Nov 02 22:17:16.378233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.378214 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462435 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-podinfo\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462476 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-data\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462506 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istiod-ca-cert\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462774 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462675 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-envoy\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462835 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462809 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-token\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462878 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462847 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-certs\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462878 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462870 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-socket\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.462975 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462899 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-credential-socket\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.463026 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.462954 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9c58\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-kube-api-access-w9c58\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.471386 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.471364 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d16d3a-openshift-default-dockercfg-rg4lf\"" Nov 02 22:17:16.489440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.489412 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9"] Nov 02 22:17:16.493263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.493235 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:16.564189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564156 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-envoy\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564259 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-token\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564288 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-certs\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564315 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-socket\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564360 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-credential-socket\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564388 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9c58\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-kube-api-access-w9c58\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564726 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564454 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-podinfo\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564726 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564483 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-data\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564726 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564515 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istiod-ca-cert\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564894 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564740 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-certs\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.564894 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564751 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-credential-socket\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.565069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564909 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-socket\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.565069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.564958 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-data\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.565372 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.565350 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istiod-ca-cert\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.566560 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.566544 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-envoy\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.567004 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.566953 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-podinfo\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.619473 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.619445 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-token\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.619730 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.619709 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9c58\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-kube-api-access-w9c58\") pod \"e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.690881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.690820 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:16.696355 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:16.696325 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:17.116880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.116849 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9"] Nov 02 22:17:17.117308 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:17:17.117172 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c20dfc_cfb5_47f5_98cf_23cb6259cfd1.slice/crio-e8317b540e19f17d0e9436f2199457b64f626cbd755f61fba64763bf11cc18a3 WatchSource:0}: Error finding container e8317b540e19f17d0e9436f2199457b64f626cbd755f61fba64763bf11cc18a3: Status 404 returned error can't find the container with id e8317b540e19f17d0e9436f2199457b64f626cbd755f61fba64763bf11cc18a3 Nov 02 22:17:17.119314 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.119283 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:17.119425 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.119339 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:17.119425 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.119373 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:17.361157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.361119 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" event={"ID":"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1","Type":"ContainerStarted","Data":"01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac"} Nov 02 22:17:17.361337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.361181 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" event={"ID":"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1","Type":"ContainerStarted","Data":"e8317b540e19f17d0e9436f2199457b64f626cbd755f61fba64763bf11cc18a3"} Nov 02 22:17:17.471665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.471558 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" podStartSLOduration=1.47153903 podStartE2EDuration="1.47153903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:17.469806105 +0000 UTC m=+3867.025434460" watchObservedRunningTime="2026-11-02 22:17:17.47153903 +0000 UTC m=+3867.027167368" Nov 02 22:17:17.691052 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:17.691021 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:18.697451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:18.697422 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:19.372227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:19.372198 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:19.373068 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:19.373050 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:20.212303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:20.212270 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:20.383350 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:20.383312 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:20.511336 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:20.511266 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:23.316723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.316688 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:23.477040 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.477003 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:23.726070 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.725982 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k"] Nov 02 22:17:23.729773 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.729753 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:23.782639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.782601 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k"] Nov 02 22:17:23.825429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.825392 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8f13dc74-dockercfg-dgwzf\"" Nov 02 22:17:23.825788 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.825762 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d16d3a\"" Nov 02 22:17:23.840329 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.840289 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnjs\" (UniqueName: \"kubernetes.io/projected/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-kube-api-access-ltnjs\") pod \"payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:23.840600 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.840375 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:23.890170 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.890137 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp"] Nov 02 22:17:23.893922 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.893897 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:23.941131 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.941097 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnjs\" (UniqueName: \"kubernetes.io/projected/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-kube-api-access-ltnjs\") pod \"payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:23.941277 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.941141 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhks5\" (UniqueName: \"kubernetes.io/projected/08d04b8c-585a-4444-9328-6a875876a89c-kube-api-access-vhks5\") pod \"payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:23.941277 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.941257 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d04b8c-585a-4444-9328-6a875876a89c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:23.941404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.941384 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:23.942141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.942113 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:23.946769 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.946745 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp"] Nov 02 22:17:23.994414 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:23.994358 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnjs\" (UniqueName: \"kubernetes.io/projected/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-kube-api-access-ltnjs\") pod \"payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:24.042873 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.042843 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhks5\" (UniqueName: \"kubernetes.io/projected/08d04b8c-585a-4444-9328-6a875876a89c-kube-api-access-vhks5\") pod \"payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:24.043022 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.042941 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:24.043022 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.042901 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d04b8c-585a-4444-9328-6a875876a89c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:24.043566 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.043548 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d04b8c-585a-4444-9328-6a875876a89c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:24.098254 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.098223 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhks5\" (UniqueName: \"kubernetes.io/projected/08d04b8c-585a-4444-9328-6a875876a89c-kube-api-access-vhks5\") pod \"payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:24.207664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.207634 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:24.407919 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.407889 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k"] Nov 02 22:17:24.407919 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:17:24.407899 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229d13b1_b5fe_4cfa_8e66_b1bdafa9057a.slice/crio-057454b831d5d054f12388f2f43c6f37eee0e2317c197eff972b801d2f75ac18 WatchSource:0}: Error finding container 057454b831d5d054f12388f2f43c6f37eee0e2317c197eff972b801d2f75ac18: Status 404 returned error can't find the container with id 057454b831d5d054f12388f2f43c6f37eee0e2317c197eff972b801d2f75ac18 Nov 02 22:17:24.570481 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:17:24.570449 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d04b8c_585a_4444_9328_6a875876a89c.slice/crio-73a6d30cb1315c2856a4a656fba540b43394bddada374b6b90deb306a97192aa WatchSource:0}: Error finding container 73a6d30cb1315c2856a4a656fba540b43394bddada374b6b90deb306a97192aa: Status 404 returned error can't find the container with id 73a6d30cb1315c2856a4a656fba540b43394bddada374b6b90deb306a97192aa Nov 02 22:17:24.570630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:24.570516 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp"] Nov 02 22:17:25.410657 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.409570 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" event={"ID":"08d04b8c-585a-4444-9328-6a875876a89c","Type":"ContainerStarted","Data":"354ceb854f92ab18f71a2613f221ade34fd4ee3d386048533ce67266ab1525e7"} Nov 02 22:17:25.410657 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.409643 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" event={"ID":"08d04b8c-585a-4444-9328-6a875876a89c","Type":"ContainerStarted","Data":"73a6d30cb1315c2856a4a656fba540b43394bddada374b6b90deb306a97192aa"} Nov 02 22:17:25.410657 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.410579 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:25.414810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.414781 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" event={"ID":"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a","Type":"ContainerStarted","Data":"69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988"} Nov 02 22:17:25.415010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.414826 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" event={"ID":"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a","Type":"ContainerStarted","Data":"057454b831d5d054f12388f2f43c6f37eee0e2317c197eff972b801d2f75ac18"} Nov 02 22:17:25.415010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.414874 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:25.520307 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.520261 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" podStartSLOduration=2.5202485550000002 podStartE2EDuration="2.520248555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:25.516650419 +0000 UTC m=+3875.072278756" watchObservedRunningTime="2026-11-02 22:17:25.520248555 +0000 UTC m=+3875.075876892" Nov 02 22:17:25.632009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:25.626924 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" podStartSLOduration=2.626904391 podStartE2EDuration="2.626904391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:25.622941522 +0000 UTC m=+3875.178569859" watchObservedRunningTime="2026-11-02 22:17:25.626904391 +0000 UTC m=+3875.182532729" Nov 02 22:17:26.002833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:26.002799 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:26.162294 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:26.162262 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:26.279220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:26.279140 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:30.741642 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:30.741605 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:35.497206 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:35.497141 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:36.043553 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:36.043518 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph"] Nov 02 22:17:36.095819 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:36.095779 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bwbph"] Nov 02 22:17:36.423831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:36.423798 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:17:36.602075 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:36.602044 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad7b3ce-6375-4f34-b3df-6d1331299f1b" path="/var/lib/kubelet/pods/7ad7b3ce-6375-4f34-b3df-6d1331299f1b/volumes" Nov 02 22:17:37.429982 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:37.429937 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:17:40.083779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:40.083749 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:40.613816 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:17:40.613778 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:17:43.548734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:43.548630 4441 generic.go:356] "Generic (PLEG): container finished" podID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerID="01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac" exitCode=0 Nov 02 22:17:43.548734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:43.548701 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" event={"ID":"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1","Type":"ContainerDied","Data":"01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac"} Nov 02 22:17:43.549308 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:43.549148 4441 scope.go:122] "RemoveContainer" containerID="01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac" Nov 02 22:17:43.550305 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:43.550276 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:43.550438 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:43.550356 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:43.550438 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:43.550404 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:44.557005 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:44.556948 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" event={"ID":"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1","Type":"ContainerStarted","Data":"3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec"} Nov 02 22:17:44.691989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:44.691393 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:44.691989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:44.691543 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 02 22:17:44.691989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:44.691585 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 02 22:17:44.956153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:44.956071 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:45.164042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:45.164010 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:45.567443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:45.567412 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:45.694374 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:45.694341 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 02 22:17:45.694557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:45.694405 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 02 22:17:45.697172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:45.697139 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:45.821081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:45.820994 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:46.691497 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:46.691450 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 02 22:17:46.692565 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:46.692126 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 02 22:17:47.693368 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:47.693329 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 02 22:17:47.694111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:47.694068 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 02 22:17:48.047643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.047561 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk"] Nov 02 22:17:48.097491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.097448 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4v8mk"] Nov 02 22:17:48.220772 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.220737 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt"] Nov 02 22:17:48.225403 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.225371 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.298813 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.298733 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d16d3a\"/\"default-dockercfg-ff7cb\"" Nov 02 22:17:48.300437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.300411 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt"] Nov 02 22:17:48.309406 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.309381 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:48.336603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.336579 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-home\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.336702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.336619 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-tmp-dir\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.336702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.336649 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-model-cache\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.336802 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.336779 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-dshm\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.336901 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.336879 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8f9\" (UniqueName: \"kubernetes.io/projected/7dcbfa37-ce0a-4565-82d9-49754cec667b-kube-api-access-fp8f9\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.337000 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.336985 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dcbfa37-ce0a-4565-82d9-49754cec667b-tls-certs\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.338572 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.338551 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d16d3a\"/\"openshift-service-ca.crt\"" Nov 02 22:17:48.338572 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.338562 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d16d3a\"/\"simulator-single-main-d16d3a-kserve-self-signed-certs\"" Nov 02 22:17:48.338692 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.338570 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d16d3a\"/\"kube-root-ca.crt\"" Nov 02 22:17:48.437846 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.437801 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-dshm\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.438107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.437867 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8f9\" (UniqueName: \"kubernetes.io/projected/7dcbfa37-ce0a-4565-82d9-49754cec667b-kube-api-access-fp8f9\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.438107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.437928 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dcbfa37-ce0a-4565-82d9-49754cec667b-tls-certs\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.438107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.438018 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-home\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.438107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.438047 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-tmp-dir\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.438107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.438064 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-model-cache\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.439202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.439164 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-model-cache\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.439362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.439337 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-home\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.439505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.439487 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-tmp-dir\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.440440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.440410 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-dshm\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.440702 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.440685 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dcbfa37-ce0a-4565-82d9-49754cec667b-tls-certs\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.496581 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.496548 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8f9\" (UniqueName: \"kubernetes.io/projected/7dcbfa37-ce0a-4565-82d9-49754cec667b-kube-api-access-fp8f9\") pod \"simulator-single-main-d16d3a-kserve-c5864d8dc-569xt\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.539278 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.539247 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:48.606363 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.606315 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763" path="/var/lib/kubelet/pods/4fe63e54-2dea-4ddf-90b3-1a4a2f6bb763/volumes" Nov 02 22:17:48.693993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.693345 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 02 22:17:48.693993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.693445 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 02 22:17:48.939448 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:17:48.939419 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dcbfa37_ce0a_4565_82d9_49754cec667b.slice/crio-892e377ce45b99fe110786e58e5367c50591916333eb9e5271f02121cac7d4e1 WatchSource:0}: Error finding container 892e377ce45b99fe110786e58e5367c50591916333eb9e5271f02121cac7d4e1: Status 404 returned error can't find the container with id 892e377ce45b99fe110786e58e5367c50591916333eb9e5271f02121cac7d4e1 Nov 02 22:17:48.939568 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:48.939547 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt"] Nov 02 22:17:49.377667 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:49.377616 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:49.614934 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:49.614895 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" event={"ID":"7dcbfa37-ce0a-4565-82d9-49754cec667b","Type":"ContainerStarted","Data":"782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d"} Nov 02 22:17:49.615163 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:49.614942 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" event={"ID":"7dcbfa37-ce0a-4565-82d9-49754cec667b","Type":"ContainerStarted","Data":"892e377ce45b99fe110786e58e5367c50591916333eb9e5271f02121cac7d4e1"} Nov 02 22:17:49.679794 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:49.679721 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:49.695139 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:49.695117 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:49.696009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:49.695992 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:17:49.723210 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:49.723141 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" podStartSLOduration=1.72312021 podStartE2EDuration="1.72312021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:49.721870573 +0000 UTC m=+3899.277498911" watchObservedRunningTime="2026-11-02 22:17:49.72312021 +0000 UTC m=+3899.278748549" Nov 02 22:17:50.539665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:50.539635 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:51.797844 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:51.797816 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:17:51.806870 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:51.806845 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:17:51.807168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:51.807150 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:17:51.814932 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:51.814901 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:17:51.818079 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:51.818060 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:17:51.825636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:51.825618 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:17:54.283660 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:54.283629 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:58.540154 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:58.540111 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:58.553110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:58.553084 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:58.678939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:58.678909 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:17:58.760077 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:17:58.760045 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:02.337260 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:02.337179 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:03.466742 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:03.466707 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:07.869188 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:07.869154 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:10.615709 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:18:10.615673 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:18:12.349577 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:12.349543 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:16.862159 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:16.862122 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:21.455943 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:21.455908 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:25.946093 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:25.946065 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:27.499036 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.499002 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48"] Nov 02 22:18:27.502674 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.502653 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.506541 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.506516 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.506657 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.506582 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwc8\" (UniqueName: \"kubernetes.io/projected/8b80291d-a49f-405c-9b5e-b7c8d0e48085-kube-api-access-tjwc8\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.506712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.506668 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.554992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.554941 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48"] Nov 02 22:18:27.598353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.598322 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:18:27.598353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.598349 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:18:27.598518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.598388 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:18:27.600239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.600225 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:18:27.607679 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.607656 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.607805 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.607712 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.607805 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.607791 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwc8\" (UniqueName: \"kubernetes.io/projected/8b80291d-a49f-405c-9b5e-b7c8d0e48085-kube-api-access-tjwc8\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.608378 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.608360 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.610201 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.610180 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.662072 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.662046 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwc8\" (UniqueName: \"kubernetes.io/projected/8b80291d-a49f-405c-9b5e-b7c8d0e48085-kube-api-access-tjwc8\") pod \"maas-api-revoke-sub-319097dc-r4p48\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:27.815070 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:27.815048 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:28.182689 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:28.182656 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48"] Nov 02 22:18:28.183347 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:18:28.183320 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b80291d_a49f_405c_9b5e_b7c8d0e48085.slice/crio-9208b8bd9606667dcb549d38ace6127b8c4d711715d51d08cf921ffdff07cbc1 WatchSource:0}: Error finding container 9208b8bd9606667dcb549d38ace6127b8c4d711715d51d08cf921ffdff07cbc1: Status 404 returned error can't find the container with id 9208b8bd9606667dcb549d38ace6127b8c4d711715d51d08cf921ffdff07cbc1 Nov 02 22:18:28.860487 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:28.860449 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" event={"ID":"8b80291d-a49f-405c-9b5e-b7c8d0e48085","Type":"ContainerStarted","Data":"93e22e881e0f814a3e3b59d511d86bf4b19bc3e7cf13df61b2ac6f2f81c9cb4f"} Nov 02 22:18:28.860923 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:28.860543 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" event={"ID":"8b80291d-a49f-405c-9b5e-b7c8d0e48085","Type":"ContainerStarted","Data":"9208b8bd9606667dcb549d38ace6127b8c4d711715d51d08cf921ffdff07cbc1"} Nov 02 22:18:28.969028 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:28.968956 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" podStartSLOduration=1.96894233 podStartE2EDuration="1.96894233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:28.967641863 +0000 UTC m=+3938.523270205" watchObservedRunningTime="2026-11-02 22:18:28.96894233 +0000 UTC m=+3938.524570667" Nov 02 22:18:29.868165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:29.868133 4441 generic.go:356] "Generic (PLEG): container finished" podID="8b80291d-a49f-405c-9b5e-b7c8d0e48085" containerID="93e22e881e0f814a3e3b59d511d86bf4b19bc3e7cf13df61b2ac6f2f81c9cb4f" exitCode=0 Nov 02 22:18:29.868566 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:29.868227 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" event={"ID":"8b80291d-a49f-405c-9b5e-b7c8d0e48085","Type":"ContainerDied","Data":"93e22e881e0f814a3e3b59d511d86bf4b19bc3e7cf13df61b2ac6f2f81c9cb4f"} Nov 02 22:18:30.456392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:30.456354 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:31.003218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.003190 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:31.042598 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.042558 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-xks-ca\") pod \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " Nov 02 22:18:31.042793 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.042621 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b80291d-a49f-405c-9b5e-b7c8d0e48085-kube-api-access-tjwc8\" (UniqueName: \"kubernetes.io/projected/8b80291d-a49f-405c-9b5e-b7c8d0e48085-kube-api-access-tjwc8\") pod \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " Nov 02 22:18:31.042862 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.042815 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-openshift-ca\") pod \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\" (UID: \"8b80291d-a49f-405c-9b5e-b7c8d0e48085\") " Nov 02 22:18:31.043187 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.043163 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-openshift-ca" pod "8b80291d-a49f-405c-9b5e-b7c8d0e48085" (UID: "8b80291d-a49f-405c-9b5e-b7c8d0e48085"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:31.044752 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.044722 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-xks-ca" pod "8b80291d-a49f-405c-9b5e-b7c8d0e48085" (UID: "8b80291d-a49f-405c-9b5e-b7c8d0e48085"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:31.044836 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.044805 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b80291d-a49f-405c-9b5e-b7c8d0e48085-kube-api-access-tjwc8" pod "8b80291d-a49f-405c-9b5e-b7c8d0e48085" (UID: "8b80291d-a49f-405c-9b5e-b7c8d0e48085"). InnerVolumeSpecName "kube-api-access-tjwc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:31.144106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.144031 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:31.144106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.144059 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b80291d-a49f-405c-9b5e-b7c8d0e48085-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:31.144106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.144069 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjwc8\" (UniqueName: \"kubernetes.io/projected/8b80291d-a49f-405c-9b5e-b7c8d0e48085-kube-api-access-tjwc8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:31.883456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.883423 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" Nov 02 22:18:31.883456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.883441 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48" event={"ID":"8b80291d-a49f-405c-9b5e-b7c8d0e48085","Type":"ContainerDied","Data":"9208b8bd9606667dcb549d38ace6127b8c4d711715d51d08cf921ffdff07cbc1"} Nov 02 22:18:31.883676 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:31.883486 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9208b8bd9606667dcb549d38ace6127b8c4d711715d51d08cf921ffdff07cbc1" Nov 02 22:18:34.043506 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:34.043471 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs"] Nov 02 22:18:34.090664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:34.090623 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-h7nfs"] Nov 02 22:18:34.582750 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:34.582717 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f7d8ee-53f8-490e-b411-682873f59856" path="/var/lib/kubelet/pods/39f7d8ee-53f8-490e-b411-682873f59856/volumes" Nov 02 22:18:34.935369 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:34.935258 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:39.177665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.177632 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:39.497809 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.497692 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:39.627512 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.627477 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp"] Nov 02 22:18:39.628125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.628106 4441 cpu_manager.go:414] "containerMap: removing container" podUID="8b80291d-a49f-405c-9b5e-b7c8d0e48085" containerName="revoke-keys" Nov 02 22:18:39.628178 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.628128 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b80291d-a49f-405c-9b5e-b7c8d0e48085" containerName="revoke-keys" Nov 02 22:18:39.628239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.628228 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b80291d-a49f-405c-9b5e-b7c8d0e48085" containerName="revoke-keys" Nov 02 22:18:39.631871 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.631847 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.684146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.684116 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp"] Nov 02 22:18:39.723020 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.722961 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhlds\" (UniqueName: \"kubernetes.io/projected/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-kube-api-access-dhlds\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.723171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.723124 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.723221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.723170 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.727011 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.726991 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:18:39.727011 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.726996 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:18:39.727133 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.727055 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:18:39.728170 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.728156 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:18:39.824071 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.824033 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlds\" (UniqueName: \"kubernetes.io/projected/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-kube-api-access-dhlds\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.824202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.824119 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.824202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.824158 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.824867 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.824843 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.826472 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.826451 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.877473 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.877450 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlds\" (UniqueName: \"kubernetes.io/projected/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-kube-api-access-dhlds\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:39.947617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:39.947593 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:40.312108 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:18:40.312072 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3de9c7_d58e_47cd_8a3a_c0e66706f2e5.slice/crio-0a73a0834505a119f612dd8d1308395c4e47a34c97eb78040d2409eaef5f298c WatchSource:0}: Error finding container 0a73a0834505a119f612dd8d1308395c4e47a34c97eb78040d2409eaef5f298c: Status 404 returned error can't find the container with id 0a73a0834505a119f612dd8d1308395c4e47a34c97eb78040d2409eaef5f298c Nov 02 22:18:40.312496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:40.312139 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp"] Nov 02 22:18:40.613915 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:18:40.613877 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:18:40.947224 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:40.947133 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" event={"ID":"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5","Type":"ContainerStarted","Data":"d37758fd798e9e3f502b48f180066f8f0dc17b194a685235b4a9b83fdcc4ddf1"} Nov 02 22:18:40.947224 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:40.947218 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" event={"ID":"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5","Type":"ContainerStarted","Data":"0a73a0834505a119f612dd8d1308395c4e47a34c97eb78040d2409eaef5f298c"} Nov 02 22:18:41.056097 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:41.056047 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" podStartSLOduration=2.056033157 podStartE2EDuration="2.056033157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:41.054004883 +0000 UTC m=+3950.609633217" watchObservedRunningTime="2026-11-02 22:18:41.056033157 +0000 UTC m=+3950.611661511" Nov 02 22:18:41.955138 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:41.955102 4441 generic.go:356] "Generic (PLEG): container finished" podID="ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" containerID="d37758fd798e9e3f502b48f180066f8f0dc17b194a685235b4a9b83fdcc4ddf1" exitCode=0 Nov 02 22:18:41.955600 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:41.955185 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" event={"ID":"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5","Type":"ContainerDied","Data":"d37758fd798e9e3f502b48f180066f8f0dc17b194a685235b4a9b83fdcc4ddf1"} Nov 02 22:18:43.096670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.096640 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:43.157878 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.157840 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-openshift-ca\") pod \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " Nov 02 22:18:43.158090 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.157946 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-kube-api-access-dhlds\" (UniqueName: \"kubernetes.io/projected/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-kube-api-access-dhlds\") pod \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " Nov 02 22:18:43.158090 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.158011 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-xks-ca\") pod \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\" (UID: \"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5\") " Nov 02 22:18:43.158429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.158372 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-openshift-ca" pod "ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" (UID: "ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:43.160226 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.160203 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-xks-ca" pod "ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" (UID: "ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:43.160381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.160363 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-kube-api-access-dhlds" pod "ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" (UID: "ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5"). InnerVolumeSpecName "kube-api-access-dhlds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:43.259145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.259058 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:43.259145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.259091 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhlds\" (UniqueName: \"kubernetes.io/projected/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-kube-api-access-dhlds\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:43.259145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.259105 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:43.968688 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.968657 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" Nov 02 22:18:43.968866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.968688 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp" event={"ID":"ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5","Type":"ContainerDied","Data":"0a73a0834505a119f612dd8d1308395c4e47a34c97eb78040d2409eaef5f298c"} Nov 02 22:18:43.968866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:43.968735 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a73a0834505a119f612dd8d1308395c4e47a34c97eb78040d2409eaef5f298c" Nov 02 22:18:44.136986 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:44.136713 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt"] Nov 02 22:18:44.228261 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:44.228174 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:44.547081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:44.546995 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:48.961840 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:48.961803 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:51.028415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:51.028384 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp"] Nov 02 22:18:51.128942 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:51.128911 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k"] Nov 02 22:18:51.366133 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:51.366100 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:51.541440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:51.541403 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:53.448107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:53.448060 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:55.716717 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:55.716681 4441 scope.go:122] "RemoveContainer" containerID="d1ddeca7e0d21c8531960efe3ab4798d08abbb222e54251e94cbb7e2fd216bcf" Nov 02 22:18:55.726247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:55.726217 4441 scope.go:122] "RemoveContainer" containerID="6fd6a2ed7c012738609e68cd70df490edcdfa530eab1e5362c145d0dcdd93f1f" Nov 02 22:18:56.029201 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:56.029098 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" podUID="08d04b8c-585a-4444-9328-6a875876a89c" containerName="payload-processing" containerID="cri-o://354ceb854f92ab18f71a2613f221ade34fd4ee3d386048533ce67266ab1525e7" gracePeriod=25 Nov 02 22:18:56.129532 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:56.129495 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" podUID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" containerName="payload-pre-processing" containerID="cri-o://69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988" gracePeriod=25 Nov 02 22:18:56.422030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:56.421990 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" podUID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:18:57.399146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:57.399111 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:57.428135 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:57.428101 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" podUID="08d04b8c-585a-4444-9328-6a875876a89c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:18:58.805793 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:58.805756 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:59.063757 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.063721 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:59.199845 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.199806 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" podUID="7dcbfa37-ce0a-4565-82d9-49754cec667b" containerName="main" containerID="cri-o://782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d" gracePeriod=45 Nov 02 22:18:59.397649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.397608 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp"] Nov 02 22:18:59.442669 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.442643 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d16d3a-78rnp"] Nov 02 22:18:59.456313 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.456288 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:18:59.625204 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625124 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-dshm\") pod \"7dcbfa37-ce0a-4565-82d9-49754cec667b\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " Nov 02 22:18:59.625204 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625193 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-model-cache\") pod \"7dcbfa37-ce0a-4565-82d9-49754cec667b\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " Nov 02 22:18:59.625415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625251 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcbfa37-ce0a-4565-82d9-49754cec667b-kube-api-access-fp8f9\" (UniqueName: \"kubernetes.io/projected/7dcbfa37-ce0a-4565-82d9-49754cec667b-kube-api-access-fp8f9\") pod \"7dcbfa37-ce0a-4565-82d9-49754cec667b\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " Nov 02 22:18:59.625415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625284 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcbfa37-ce0a-4565-82d9-49754cec667b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dcbfa37-ce0a-4565-82d9-49754cec667b-tls-certs\") pod \"7dcbfa37-ce0a-4565-82d9-49754cec667b\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " Nov 02 22:18:59.625415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625334 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-tmp-dir\") pod \"7dcbfa37-ce0a-4565-82d9-49754cec667b\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " Nov 02 22:18:59.625583 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625431 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-model-cache" pod "7dcbfa37-ce0a-4565-82d9-49754cec667b" (UID: "7dcbfa37-ce0a-4565-82d9-49754cec667b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:59.625583 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625454 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-home\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-home\") pod \"7dcbfa37-ce0a-4565-82d9-49754cec667b\" (UID: \"7dcbfa37-ce0a-4565-82d9-49754cec667b\") " Nov 02 22:18:59.625865 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625842 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:59.625865 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625844 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-home" pod "7dcbfa37-ce0a-4565-82d9-49754cec667b" (UID: "7dcbfa37-ce0a-4565-82d9-49754cec667b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:59.626048 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.625867 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-tmp-dir" pod "7dcbfa37-ce0a-4565-82d9-49754cec667b" (UID: "7dcbfa37-ce0a-4565-82d9-49754cec667b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:59.627526 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.627500 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-dshm" pod "7dcbfa37-ce0a-4565-82d9-49754cec667b" (UID: "7dcbfa37-ce0a-4565-82d9-49754cec667b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:59.627658 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.627645 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcbfa37-ce0a-4565-82d9-49754cec667b-tls-certs" pod "7dcbfa37-ce0a-4565-82d9-49754cec667b" (UID: "7dcbfa37-ce0a-4565-82d9-49754cec667b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:59.628050 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.628029 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcbfa37-ce0a-4565-82d9-49754cec667b-kube-api-access-fp8f9" pod "7dcbfa37-ce0a-4565-82d9-49754cec667b" (UID: "7dcbfa37-ce0a-4565-82d9-49754cec667b"). InnerVolumeSpecName "kube-api-access-fp8f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:59.726604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.726569 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp8f9\" (UniqueName: \"kubernetes.io/projected/7dcbfa37-ce0a-4565-82d9-49754cec667b-kube-api-access-fp8f9\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:59.726604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.726607 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dcbfa37-ce0a-4565-82d9-49754cec667b-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:59.726814 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.726622 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:59.726814 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.726634 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:59.726814 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.726646 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dcbfa37-ce0a-4565-82d9-49754cec667b-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:59.792056 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:18:59.792024 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:00.067437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.067401 4441 generic.go:356] "Generic (PLEG): container finished" podID="7dcbfa37-ce0a-4565-82d9-49754cec667b" containerID="782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d" exitCode=0 Nov 02 22:19:00.067888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.067460 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" Nov 02 22:19:00.067888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.067485 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" event={"ID":"7dcbfa37-ce0a-4565-82d9-49754cec667b","Type":"ContainerDied","Data":"782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d"} Nov 02 22:19:00.067888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.067528 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt" event={"ID":"7dcbfa37-ce0a-4565-82d9-49754cec667b","Type":"ContainerDied","Data":"892e377ce45b99fe110786e58e5367c50591916333eb9e5271f02121cac7d4e1"} Nov 02 22:19:00.067888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.067545 4441 scope.go:122] "RemoveContainer" containerID="782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d" Nov 02 22:19:00.077865 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.077846 4441 scope.go:122] "RemoveContainer" containerID="782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d" Nov 02 22:19:00.078199 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:19:00.078180 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d\": container with ID starting with 782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d not found: ID does not exist" containerID="782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d" Nov 02 22:19:00.078261 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.078206 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d"} err="failed to get container status \"782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d\": rpc error: code = NotFound desc = could not find container \"782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d\": container with ID starting with 782adc3509dce9c3f680581f25953954c4c5c290db6d28cbd4e6d6246f83ec1d not found: ID does not exist" Nov 02 22:19:00.223871 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.223834 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt"] Nov 02 22:19:00.270907 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.270877 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d16d3a/simulator-single-main-d16d3a-kserve-c5864d8dc-569xt"] Nov 02 22:19:00.582694 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.582657 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcbfa37-ce0a-4565-82d9-49754cec667b" path="/var/lib/kubelet/pods/7dcbfa37-ce0a-4565-82d9-49754cec667b/volumes" Nov 02 22:19:00.583149 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:00.583129 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" path="/var/lib/kubelet/pods/ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5/volumes" Nov 02 22:19:02.710334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:02.710303 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:05.752414 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:05.752373 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:05.881223 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:05.881190 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:06.067475 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:06.067445 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:06.074536 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:06.074510 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9"] Nov 02 22:19:06.074744 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:06.074723 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" containerID="cri-o://3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec" gracePeriod=30 Nov 02 22:19:06.422164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:06.422080 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" podUID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:19:07.427879 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:07.427839 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" podUID="08d04b8c-585a-4444-9328-6a875876a89c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:19:07.620090 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:07.620059 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6"] Nov 02 22:19:07.665405 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:07.665368 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xvpk6"] Nov 02 22:19:08.043736 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:08.043703 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g"] Nov 02 22:19:08.090558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:08.090524 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gcr2g"] Nov 02 22:19:08.582395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:08.582354 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5af569-a2c4-453a-b261-6ca0033b8da1" path="/var/lib/kubelet/pods/6f5af569-a2c4-453a-b261-6ca0033b8da1/volumes" Nov 02 22:19:08.583009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:08.582963 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca155641-0b79-4498-806b-7bc00e8fce36" path="/var/lib/kubelet/pods/ca155641-0b79-4498-806b-7bc00e8fce36/volumes" Nov 02 22:19:10.614928 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:19:10.614892 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:19:11.330079 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.330056 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:19:11.436756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.436677 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-token\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-token\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.436756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.436742 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-data\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.436798 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-certs\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.436845 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-socket\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.436883 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-envoy\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.436945 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istiod-ca-cert\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437040 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-kube-api-access-w9c58\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-kube-api-access-w9c58\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437080 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-credential-socket\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437079 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-data" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:11.437251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437125 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-podinfo\") pod \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\" (UID: \"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1\") " Nov 02 22:19:11.437251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437156 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-socket" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:11.437251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437173 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-certs" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:11.437558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437434 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.437558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437458 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.437558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437452 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-credential-socket" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:11.437558 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437474 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.437783 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.437569 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istiod-ca-cert" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:11.439156 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.439129 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-token" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:11.439273 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.439153 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-envoy" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:11.439417 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.439399 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-kube-api-access-w9c58" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "kube-api-access-w9c58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:11.439518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.439505 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-podinfo" pod "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" (UID: "18c20dfc-cfb5-47f5-98cf-23cb6259cfd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:19:11.538127 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.538090 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.538127 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.538119 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.538127 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.538130 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9c58\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-kube-api-access-w9c58\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.538374 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.538140 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.538374 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.538149 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.538374 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:11.538160 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:12.154273 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.154238 4441 generic.go:356] "Generic (PLEG): container finished" podID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerID="3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec" exitCode=0 Nov 02 22:19:12.154787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.154310 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" Nov 02 22:19:12.154787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.154317 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" event={"ID":"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1","Type":"ContainerDied","Data":"3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec"} Nov 02 22:19:12.154787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.154393 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9" event={"ID":"18c20dfc-cfb5-47f5-98cf-23cb6259cfd1","Type":"ContainerDied","Data":"e8317b540e19f17d0e9436f2199457b64f626cbd755f61fba64763bf11cc18a3"} Nov 02 22:19:12.154787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.154416 4441 scope.go:122] "RemoveContainer" containerID="3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec" Nov 02 22:19:12.164761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.164736 4441 scope.go:122] "RemoveContainer" containerID="01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac" Nov 02 22:19:12.173229 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.173212 4441 scope.go:122] "RemoveContainer" containerID="3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec" Nov 02 22:19:12.173472 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:19:12.173456 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec\": container with ID starting with 3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec not found: ID does not exist" containerID="3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec" Nov 02 22:19:12.173525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.173481 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec"} err="failed to get container status \"3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec\": rpc error: code = NotFound desc = could not find container \"3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec\": container with ID starting with 3cbe0d1ffa0d4861ba465fd10b114f66647c64a89cca6c8f5b86b8c34eea6bec not found: ID does not exist" Nov 02 22:19:12.173525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.173496 4441 scope.go:122] "RemoveContainer" containerID="01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac" Nov 02 22:19:12.173732 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:19:12.173715 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac\": container with ID starting with 01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac not found: ID does not exist" containerID="01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac" Nov 02 22:19:12.173769 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.173737 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac"} err="failed to get container status \"01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac\": rpc error: code = NotFound desc = could not find container \"01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac\": container with ID starting with 01c25c09769d68009e41e3c95e01b3564e04444f7eb49ae064862d11791540ac not found: ID does not exist" Nov 02 22:19:12.357663 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.357633 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9"] Nov 02 22:19:12.404783 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.404714 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d16d3a-openshift-default-784968dbdft7lb9"] Nov 02 22:19:12.583315 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:12.583282 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" path="/var/lib/kubelet/pods/18c20dfc-cfb5-47f5-98cf-23cb6259cfd1/volumes" Nov 02 22:19:13.645455 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:13.645422 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:16.180890 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.180853 4441 generic.go:356] "Generic (PLEG): container finished" podID="08d04b8c-585a-4444-9328-6a875876a89c" containerID="354ceb854f92ab18f71a2613f221ade34fd4ee3d386048533ce67266ab1525e7" exitCode=0 Nov 02 22:19:16.181304 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.180915 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" event={"ID":"08d04b8c-585a-4444-9328-6a875876a89c","Type":"ContainerDied","Data":"354ceb854f92ab18f71a2613f221ade34fd4ee3d386048533ce67266ab1525e7"} Nov 02 22:19:16.293013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.292947 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:19:16.382267 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.382243 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:19:16.382560 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.382546 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d04b8c-585a-4444-9328-6a875876a89c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d04b8c-585a-4444-9328-6a875876a89c-plugins-config-volume\") pod \"08d04b8c-585a-4444-9328-6a875876a89c\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " Nov 02 22:19:16.382617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.382602 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d04b8c-585a-4444-9328-6a875876a89c-kube-api-access-vhks5\" (UniqueName: \"kubernetes.io/projected/08d04b8c-585a-4444-9328-6a875876a89c-kube-api-access-vhks5\") pod \"08d04b8c-585a-4444-9328-6a875876a89c\" (UID: \"08d04b8c-585a-4444-9328-6a875876a89c\") " Nov 02 22:19:16.382881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.382855 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d04b8c-585a-4444-9328-6a875876a89c-plugins-config-volume" pod "08d04b8c-585a-4444-9328-6a875876a89c" (UID: "08d04b8c-585a-4444-9328-6a875876a89c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:16.384495 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.384476 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d04b8c-585a-4444-9328-6a875876a89c-kube-api-access-vhks5" pod "08d04b8c-585a-4444-9328-6a875876a89c" (UID: "08d04b8c-585a-4444-9328-6a875876a89c"). InnerVolumeSpecName "kube-api-access-vhks5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:16.483989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.483935 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-kube-api-access-ltnjs\" (UniqueName: \"kubernetes.io/projected/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-kube-api-access-ltnjs\") pod \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " Nov 02 22:19:16.484168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.484077 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-plugins-config-volume\") pod \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\" (UID: \"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a\") " Nov 02 22:19:16.484270 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.484256 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhks5\" (UniqueName: \"kubernetes.io/projected/08d04b8c-585a-4444-9328-6a875876a89c-kube-api-access-vhks5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:16.484321 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.484273 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d04b8c-585a-4444-9328-6a875876a89c-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:16.484464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.484444 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-plugins-config-volume" pod "229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" (UID: "229d13b1-b5fe-4cfa-8e66-b1bdafa9057a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:16.486029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.486009 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-kube-api-access-ltnjs" pod "229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" (UID: "229d13b1-b5fe-4cfa-8e66-b1bdafa9057a"). InnerVolumeSpecName "kube-api-access-ltnjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:16.585044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.585016 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:16.585044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:16.585040 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnjs\" (UniqueName: \"kubernetes.io/projected/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a-kube-api-access-ltnjs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:17.188931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.188903 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" Nov 02 22:19:17.189397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.188899 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp" event={"ID":"08d04b8c-585a-4444-9328-6a875876a89c","Type":"ContainerDied","Data":"73a6d30cb1315c2856a4a656fba540b43394bddada374b6b90deb306a97192aa"} Nov 02 22:19:17.189397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.189050 4441 scope.go:122] "RemoveContainer" containerID="354ceb854f92ab18f71a2613f221ade34fd4ee3d386048533ce67266ab1525e7" Nov 02 22:19:17.190557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.190532 4441 generic.go:356] "Generic (PLEG): container finished" podID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" containerID="69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988" exitCode=0 Nov 02 22:19:17.190662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.190571 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" Nov 02 22:19:17.190709 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.190660 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" event={"ID":"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a","Type":"ContainerDied","Data":"69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988"} Nov 02 22:19:17.190746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.190735 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k" event={"ID":"229d13b1-b5fe-4cfa-8e66-b1bdafa9057a","Type":"ContainerDied","Data":"057454b831d5d054f12388f2f43c6f37eee0e2317c197eff972b801d2f75ac18"} Nov 02 22:19:17.199518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.199502 4441 scope.go:122] "RemoveContainer" containerID="69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988" Nov 02 22:19:17.207676 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.207644 4441 scope.go:122] "RemoveContainer" containerID="69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988" Nov 02 22:19:17.216537 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:19:17.216503 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988\": container with ID starting with 69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988 not found: ID does not exist" containerID="69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988" Nov 02 22:19:17.216675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.216546 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988"} err="failed to get container status \"69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988\": rpc error: code = NotFound desc = could not find container \"69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988\": container with ID starting with 69c72dc3944e2bf05f40150bf12ef04c43d25d7a7b669574933f5872b1179988 not found: ID does not exist" Nov 02 22:19:17.343656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.343630 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k"] Nov 02 22:19:17.346867 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.346847 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d16d3a-7b844t5d5k"] Nov 02 22:19:17.449211 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.449188 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp"] Nov 02 22:19:17.452776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:17.452752 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d16d3a-69cd7f47bb94qp"] Nov 02 22:19:18.582251 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:18.582217 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d04b8c-585a-4444-9328-6a875876a89c" path="/var/lib/kubelet/pods/08d04b8c-585a-4444-9328-6a875876a89c/volumes" Nov 02 22:19:18.582670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:18.582618 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" path="/var/lib/kubelet/pods/229d13b1-b5fe-4cfa-8e66-b1bdafa9057a/volumes" Nov 02 22:19:20.627238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:20.627208 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:21.107130 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:21.107087 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:22.569773 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:22.569738 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:22.688843 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:22.688799 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:26.626904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:26.626868 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:29.470397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.470358 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg"] Nov 02 22:19:29.470917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.470588 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" podUID="580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" containerName="manager" containerID="cri-o://ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71" gracePeriod=10 Nov 02 22:19:29.718190 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.718167 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 22:19:29.799634 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.799575 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-kube-api-access-xqddd\" (UniqueName: \"kubernetes.io/projected/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-kube-api-access-xqddd\") pod \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " Nov 02 22:19:29.799767 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.799650 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-extensions-socket-volume\") pod \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\" (UID: \"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5\") " Nov 02 22:19:29.800044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.800023 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-extensions-socket-volume" pod "580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" (UID: "580cef2d-3c2b-4b12-bfe5-d82f44cfaba5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:29.801595 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.801575 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-kube-api-access-xqddd" pod "580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" (UID: "580cef2d-3c2b-4b12-bfe5-d82f44cfaba5"). InnerVolumeSpecName "kube-api-access-xqddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:29.900580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.900536 4441 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-extensions-socket-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:29.900580 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:29.900571 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqddd\" (UniqueName: \"kubernetes.io/projected/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5-kube-api-access-xqddd\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:30.273989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.273936 4441 generic.go:356] "Generic (PLEG): container finished" podID="580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" containerID="ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71" exitCode=0 Nov 02 22:19:30.274168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.273997 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" Nov 02 22:19:30.274168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.274011 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" event={"ID":"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5","Type":"ContainerDied","Data":"ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71"} Nov 02 22:19:30.274168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.274051 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg" event={"ID":"580cef2d-3c2b-4b12-bfe5-d82f44cfaba5","Type":"ContainerDied","Data":"20ff36fdbfefca7a8b94a14087b3de4df05071d62dcf95ac357e8f4115efbfa8"} Nov 02 22:19:30.274168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.274074 4441 scope.go:122] "RemoveContainer" containerID="ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71" Nov 02 22:19:30.288054 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.288035 4441 scope.go:122] "RemoveContainer" containerID="ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71" Nov 02 22:19:30.288291 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:19:30.288268 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71\": container with ID starting with ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71 not found: ID does not exist" containerID="ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71" Nov 02 22:19:30.288343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.288300 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71"} err="failed to get container status \"ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71\": rpc error: code = NotFound desc = could not find container \"ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71\": container with ID starting with ee845d35f1c79ce51dd2a3adeb4a068126ec9b9aa15ab31ecad6e846804fec71 not found: ID does not exist" Nov 02 22:19:30.430122 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.430090 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg"] Nov 02 22:19:30.477510 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.477479 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-627xg"] Nov 02 22:19:30.582710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:30.582683 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" path="/var/lib/kubelet/pods/580cef2d-3c2b-4b12-bfe5-d82f44cfaba5/volumes" Nov 02 22:19:33.042490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:33.042413 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw"] Nov 02 22:19:33.091328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:33.091290 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-mt4rw"] Nov 02 22:19:34.582759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:34.582726 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770342b7-c7f6-4beb-a429-4e41d9610fdf" path="/var/lib/kubelet/pods/770342b7-c7f6-4beb-a429-4e41d9610fdf/volumes" Nov 02 22:19:35.372828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:35.372796 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:35.485252 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:35.485222 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:40.614410 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:19:40.614375 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:19:45.965667 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:45.965630 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:55.753002 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:19:55.752949 4441 scope.go:122] "RemoveContainer" containerID="dcca7e1b01c74a003f65a597d9ad951a5fb11733b657bf5fd92a98ca60a5af58" Nov 02 22:20:02.994319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:02.994286 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:04.042985 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:04.042945 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc"] Nov 02 22:20:04.089862 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:04.089831 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gnfwc"] Nov 02 22:20:04.582888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:04.582855 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48314b43-3313-42d0-8681-034df736a477" path="/var/lib/kubelet/pods/48314b43-3313-42d0-8681-034df736a477/volumes" Nov 02 22:20:10.613918 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:20:10.613882 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:20:27.045390 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:27.045350 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6"] Nov 02 22:20:27.092505 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:27.092470 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-b6dc6"] Nov 02 22:20:28.582944 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:28.582903 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96666b63-4e41-4bbf-a3b8-3c0350f8de7b" path="/var/lib/kubelet/pods/96666b63-4e41-4bbf-a3b8-3c0350f8de7b/volumes" Nov 02 22:20:33.003235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003202 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c"] Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003556 4441 cpu_manager.go:414] "containerMap: removing container" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003568 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003577 4441 cpu_manager.go:414] "containerMap: removing container" podUID="08d04b8c-585a-4444-9328-6a875876a89c" containerName="payload-processing" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003583 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d04b8c-585a-4444-9328-6a875876a89c" containerName="payload-processing" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003593 4441 cpu_manager.go:414] "containerMap: removing container" podUID="580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" containerName="manager" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003597 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" containerName="manager" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003603 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" containerName="revoke-keys" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003608 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" containerName="revoke-keys" Nov 02 22:20:33.003614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003616 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7dcbfa37-ce0a-4565-82d9-49754cec667b" containerName="main" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003629 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcbfa37-ce0a-4565-82d9-49754cec667b" containerName="main" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003648 4441 cpu_manager.go:414] "containerMap: removing container" podUID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" containerName="payload-pre-processing" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003652 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" containerName="payload-pre-processing" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003661 4441 cpu_manager.go:414] "containerMap: removing container" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003665 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003716 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003724 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d04b8c-585a-4444-9328-6a875876a89c" containerName="payload-processing" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003729 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="580cef2d-3c2b-4b12-bfe5-d82f44cfaba5" containerName="manager" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003734 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3de9c7-d58e-47cd-8a3a-c0e66706f2e5" containerName="revoke-keys" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003740 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcbfa37-ce0a-4565-82d9-49754cec667b" containerName="main" Nov 02 22:20:33.003917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.003750 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="229d13b1-b5fe-4cfa-8e66-b1bdafa9057a" containerName="payload-pre-processing" Nov 02 22:20:33.007010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.006989 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.057312 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.057280 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c"] Nov 02 22:20:33.100899 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.100863 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v8rj6\"" Nov 02 22:20:33.127911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.127889 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6298a9da-6f99-4b43-93d0-7082cd03311b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5qm9c\" (UID: \"6298a9da-6f99-4b43-93d0-7082cd03311b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.128038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.127917 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpwt\" (UniqueName: \"kubernetes.io/projected/6298a9da-6f99-4b43-93d0-7082cd03311b-kube-api-access-mwpwt\") pod \"kuadrant-operator-controller-manager-678d87766f-5qm9c\" (UID: \"6298a9da-6f99-4b43-93d0-7082cd03311b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.228807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.228776 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6298a9da-6f99-4b43-93d0-7082cd03311b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5qm9c\" (UID: \"6298a9da-6f99-4b43-93d0-7082cd03311b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.228807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.228811 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpwt\" (UniqueName: \"kubernetes.io/projected/6298a9da-6f99-4b43-93d0-7082cd03311b-kube-api-access-mwpwt\") pod \"kuadrant-operator-controller-manager-678d87766f-5qm9c\" (UID: \"6298a9da-6f99-4b43-93d0-7082cd03311b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.229203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.229178 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6298a9da-6f99-4b43-93d0-7082cd03311b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5qm9c\" (UID: \"6298a9da-6f99-4b43-93d0-7082cd03311b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.283884 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.283819 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpwt\" (UniqueName: \"kubernetes.io/projected/6298a9da-6f99-4b43-93d0-7082cd03311b-kube-api-access-mwpwt\") pod \"kuadrant-operator-controller-manager-678d87766f-5qm9c\" (UID: \"6298a9da-6f99-4b43-93d0-7082cd03311b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.319777 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.319746 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:33.728252 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:33.728226 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c"] Nov 02 22:20:33.729633 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:20:33.729601 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6298a9da_6f99_4b43_93d0_7082cd03311b.slice/crio-e4c0c1b2da41682a3b049ff963fd4d27fdb474b3335e371f590d0c88d2f0b158 WatchSource:0}: Error finding container e4c0c1b2da41682a3b049ff963fd4d27fdb474b3335e371f590d0c88d2f0b158: Status 404 returned error can't find the container with id e4c0c1b2da41682a3b049ff963fd4d27fdb474b3335e371f590d0c88d2f0b158 Nov 02 22:20:34.654768 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:34.654720 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" event={"ID":"6298a9da-6f99-4b43-93d0-7082cd03311b","Type":"ContainerStarted","Data":"9f16356d67a382e9ae9c6553e554b5ece00d9b21225540e874d34cbf9302b98d"} Nov 02 22:20:34.655227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:34.654831 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:34.655227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:34.654851 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" event={"ID":"6298a9da-6f99-4b43-93d0-7082cd03311b","Type":"ContainerStarted","Data":"e4c0c1b2da41682a3b049ff963fd4d27fdb474b3335e371f590d0c88d2f0b158"} Nov 02 22:20:34.865649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:34.865588 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" podStartSLOduration=2.865571733 podStartE2EDuration="2.865571733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:34.8141192 +0000 UTC m=+4064.369747517" watchObservedRunningTime="2026-11-02 22:20:34.865571733 +0000 UTC m=+4064.421200072" Nov 02 22:20:40.616556 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:20:40.616519 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:20:45.663415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:45.663386 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5qm9c" Nov 02 22:20:55.799993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:55.799139 4441 scope.go:122] "RemoveContainer" containerID="37ade576fddec70ec8063ed360fb6c1c8486debb4976c48504afeb315c02bb0f" Nov 02 22:20:55.863311 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:55.862279 4441 scope.go:122] "RemoveContainer" containerID="0a8110a2ef3a3ba089799499d967a2cf3885a965862398c9debddcde0464d12d" Nov 02 22:20:58.720278 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.720243 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k"] Nov 02 22:20:58.720745 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.720683 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c20dfc-cfb5-47f5-98cf-23cb6259cfd1" containerName="istio-proxy" Nov 02 22:20:58.723665 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.723645 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.782571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.782540 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k"] Nov 02 22:20:58.805519 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.805489 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.805655 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.805604 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.805699 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.805662 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhscq\" (UniqueName: \"kubernetes.io/projected/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-kube-api-access-dhscq\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.821840 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.821815 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:20:58.821840 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.821816 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:58.822038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.821848 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:20:58.822038 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.821861 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:20:58.906451 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.906419 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.906602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.906456 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhscq\" (UniqueName: \"kubernetes.io/projected/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-kube-api-access-dhscq\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.906602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.906506 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.907273 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.907254 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.908883 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.908864 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:58.960410 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:58.960378 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhscq\" (UniqueName: \"kubernetes.io/projected/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-kube-api-access-dhscq\") pod \"maas-api-revoke-sub-d576e68b-xh42k\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:59.036483 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:59.036425 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:20:59.399548 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:59.399512 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k"] Nov 02 22:20:59.400945 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:20:59.400917 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2d2367_8f2e_4270_85be_4a6e7690c6fa.slice/crio-d89132c82bacd49749363f4213250af45fd6f261e1d62fa264fc9d71c17d6b86 WatchSource:0}: Error finding container d89132c82bacd49749363f4213250af45fd6f261e1d62fa264fc9d71c17d6b86: Status 404 returned error can't find the container with id d89132c82bacd49749363f4213250af45fd6f261e1d62fa264fc9d71c17d6b86 Nov 02 22:20:59.865373 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:59.865337 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" event={"ID":"7e2d2367-8f2e-4270-85be-4a6e7690c6fa","Type":"ContainerStarted","Data":"098fa65eff4a5f14c5256b95b614c867e596aed36b4b3ac6b702edac07199605"} Nov 02 22:20:59.865744 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:59.865401 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" event={"ID":"7e2d2367-8f2e-4270-85be-4a6e7690c6fa","Type":"ContainerStarted","Data":"d89132c82bacd49749363f4213250af45fd6f261e1d62fa264fc9d71c17d6b86"} Nov 02 22:20:59.975605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:20:59.975510 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" podStartSLOduration=1.975496207 podStartE2EDuration="1.975496207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:59.972376345 +0000 UTC m=+4089.528004706" watchObservedRunningTime="2026-11-02 22:20:59.975496207 +0000 UTC m=+4089.531124546" Nov 02 22:21:00.872950 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:00.872908 4441 generic.go:356] "Generic (PLEG): container finished" podID="7e2d2367-8f2e-4270-85be-4a6e7690c6fa" containerID="098fa65eff4a5f14c5256b95b614c867e596aed36b4b3ac6b702edac07199605" exitCode=0 Nov 02 22:21:00.873383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:00.873002 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" event={"ID":"7e2d2367-8f2e-4270-85be-4a6e7690c6fa","Type":"ContainerDied","Data":"098fa65eff4a5f14c5256b95b614c867e596aed36b4b3ac6b702edac07199605"} Nov 02 22:21:02.008587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.008550 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:21:02.030909 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.030879 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-openshift-ca\") pod \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " Nov 02 22:21:02.031084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.030936 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-kube-api-access-dhscq\" (UniqueName: \"kubernetes.io/projected/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-kube-api-access-dhscq\") pod \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " Nov 02 22:21:02.031297 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.031273 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-openshift-ca" pod "7e2d2367-8f2e-4270-85be-4a6e7690c6fa" (UID: "7e2d2367-8f2e-4270-85be-4a6e7690c6fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:02.032939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.032919 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-kube-api-access-dhscq" pod "7e2d2367-8f2e-4270-85be-4a6e7690c6fa" (UID: "7e2d2367-8f2e-4270-85be-4a6e7690c6fa"). InnerVolumeSpecName "kube-api-access-dhscq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:02.131900 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.131826 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-xks-ca\") pod \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\" (UID: \"7e2d2367-8f2e-4270-85be-4a6e7690c6fa\") " Nov 02 22:21:02.132091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.132033 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.132091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.132046 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhscq\" (UniqueName: \"kubernetes.io/projected/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-kube-api-access-dhscq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.134045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.134019 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-xks-ca" pod "7e2d2367-8f2e-4270-85be-4a6e7690c6fa" (UID: "7e2d2367-8f2e-4270-85be-4a6e7690c6fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:02.232793 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.232758 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2d2367-8f2e-4270-85be-4a6e7690c6fa-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.885807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.885719 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" event={"ID":"7e2d2367-8f2e-4270-85be-4a6e7690c6fa","Type":"ContainerDied","Data":"d89132c82bacd49749363f4213250af45fd6f261e1d62fa264fc9d71c17d6b86"} Nov 02 22:21:02.885807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.885769 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k" Nov 02 22:21:02.885807 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:02.885778 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d89132c82bacd49749363f4213250af45fd6f261e1d62fa264fc9d71c17d6b86" Nov 02 22:21:03.333388 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:03.333360 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:10.613751 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:21:10.613704 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:21:27.525525 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.525486 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd"] Nov 02 22:21:27.525922 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.525874 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7e2d2367-8f2e-4270-85be-4a6e7690c6fa" containerName="revoke-keys" Nov 02 22:21:27.525922 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.525885 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2d2367-8f2e-4270-85be-4a6e7690c6fa" containerName="revoke-keys" Nov 02 22:21:27.526029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.525949 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2d2367-8f2e-4270-85be-4a6e7690c6fa" containerName="revoke-keys" Nov 02 22:21:27.530326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.530301 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.581699 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.581668 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39642e40-b045-4031-943e-444bea781ea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.581892 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.581743 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjk5\" (UniqueName: \"kubernetes.io/projected/39642e40-b045-4031-943e-444bea781ea7-kube-api-access-7qjk5\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.581892 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.581842 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39642e40-b045-4031-943e-444bea781ea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.582245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.582223 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd"] Nov 02 22:21:27.625705 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.625675 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:27.625705 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.625690 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:21:27.627429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.627409 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:21:27.627532 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.627439 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:21:27.683004 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.682961 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39642e40-b045-4031-943e-444bea781ea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.683120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.683046 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjk5\" (UniqueName: \"kubernetes.io/projected/39642e40-b045-4031-943e-444bea781ea7-kube-api-access-7qjk5\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.683120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.683072 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39642e40-b045-4031-943e-444bea781ea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.683610 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.683593 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39642e40-b045-4031-943e-444bea781ea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.685419 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.685400 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39642e40-b045-4031-943e-444bea781ea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.737048 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.737029 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjk5\" (UniqueName: \"kubernetes.io/projected/39642e40-b045-4031-943e-444bea781ea7-kube-api-access-7qjk5\") pod \"maas-api-revoke-sub-d2bc93bc-b6rfd\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:27.843016 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:27.842994 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:28.207094 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:28.207063 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd"] Nov 02 22:21:28.208699 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:21:28.208664 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39642e40_b045_4031_943e_444bea781ea7.slice/crio-3e5f1b8fe4157fd03df0852dc74323d89545ba3d15f43652cfb501507a2013ff WatchSource:0}: Error finding container 3e5f1b8fe4157fd03df0852dc74323d89545ba3d15f43652cfb501507a2013ff: Status 404 returned error can't find the container with id 3e5f1b8fe4157fd03df0852dc74323d89545ba3d15f43652cfb501507a2013ff Nov 02 22:21:29.068195 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:29.068164 4441 generic.go:356] "Generic (PLEG): container finished" podID="39642e40-b045-4031-943e-444bea781ea7" containerID="eb2b7921b881fe25960140028dd8c811e8bd0592e19e38cdc34680d9ec71c5e4" exitCode=0 Nov 02 22:21:29.068530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:29.068243 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" event={"ID":"39642e40-b045-4031-943e-444bea781ea7","Type":"ContainerDied","Data":"eb2b7921b881fe25960140028dd8c811e8bd0592e19e38cdc34680d9ec71c5e4"} Nov 02 22:21:29.068530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:29.068299 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" event={"ID":"39642e40-b045-4031-943e-444bea781ea7","Type":"ContainerStarted","Data":"3e5f1b8fe4157fd03df0852dc74323d89545ba3d15f43652cfb501507a2013ff"} Nov 02 22:21:30.208395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.208372 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:30.309282 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.309257 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39642e40-b045-4031-943e-444bea781ea7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39642e40-b045-4031-943e-444bea781ea7-maas-api-xks-ca\") pod \"39642e40-b045-4031-943e-444bea781ea7\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " Nov 02 22:21:30.309423 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.309322 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39642e40-b045-4031-943e-444bea781ea7-kube-api-access-7qjk5\" (UniqueName: \"kubernetes.io/projected/39642e40-b045-4031-943e-444bea781ea7-kube-api-access-7qjk5\") pod \"39642e40-b045-4031-943e-444bea781ea7\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " Nov 02 22:21:30.309423 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.309348 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39642e40-b045-4031-943e-444bea781ea7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39642e40-b045-4031-943e-444bea781ea7-maas-api-openshift-ca\") pod \"39642e40-b045-4031-943e-444bea781ea7\" (UID: \"39642e40-b045-4031-943e-444bea781ea7\") " Nov 02 22:21:30.309792 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.309763 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39642e40-b045-4031-943e-444bea781ea7-maas-api-openshift-ca" pod "39642e40-b045-4031-943e-444bea781ea7" (UID: "39642e40-b045-4031-943e-444bea781ea7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:30.311335 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.311313 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39642e40-b045-4031-943e-444bea781ea7-maas-api-xks-ca" pod "39642e40-b045-4031-943e-444bea781ea7" (UID: "39642e40-b045-4031-943e-444bea781ea7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:30.311511 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.311491 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39642e40-b045-4031-943e-444bea781ea7-kube-api-access-7qjk5" pod "39642e40-b045-4031-943e-444bea781ea7" (UID: "39642e40-b045-4031-943e-444bea781ea7"). InnerVolumeSpecName "kube-api-access-7qjk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:30.410492 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.410429 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39642e40-b045-4031-943e-444bea781ea7-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:30.410492 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.410453 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qjk5\" (UniqueName: \"kubernetes.io/projected/39642e40-b045-4031-943e-444bea781ea7-kube-api-access-7qjk5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:30.410492 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:30.410464 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39642e40-b045-4031-943e-444bea781ea7-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:31.081617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:31.081587 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" Nov 02 22:21:31.081617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:31.081597 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd" event={"ID":"39642e40-b045-4031-943e-444bea781ea7","Type":"ContainerDied","Data":"3e5f1b8fe4157fd03df0852dc74323d89545ba3d15f43652cfb501507a2013ff"} Nov 02 22:21:31.081844 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:31.081642 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e5f1b8fe4157fd03df0852dc74323d89545ba3d15f43652cfb501507a2013ff" Nov 02 22:21:31.514277 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:31.514204 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:35.048984 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:35.048940 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv"] Nov 02 22:21:35.075678 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:35.075644 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bqwtv"] Nov 02 22:21:36.582407 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:36.582365 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dcdff7-ad05-41f1-908a-3a5556aea118" path="/var/lib/kubelet/pods/a0dcdff7-ad05-41f1-908a-3a5556aea118/volumes" Nov 02 22:21:39.884830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:39.884795 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:40.456609 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.456572 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9"] Nov 02 22:21:40.457067 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.457046 4441 cpu_manager.go:414] "containerMap: removing container" podUID="39642e40-b045-4031-943e-444bea781ea7" containerName="revoke-keys" Nov 02 22:21:40.457218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.457068 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39642e40-b045-4031-943e-444bea781ea7" containerName="revoke-keys" Nov 02 22:21:40.457218 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.457189 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="39642e40-b045-4031-943e-444bea781ea7" containerName="revoke-keys" Nov 02 22:21:40.461486 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.461464 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.544399 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.544365 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:40.556343 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.556314 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9"] Nov 02 22:21:40.599693 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.599666 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c19d19-openshift-default-dockercfg-z9gcx\"" Nov 02 22:21:40.601061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601036 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-certs\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601082 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-envoy\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601168 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-podinfo\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601295 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601226 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-credential-socket\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601295 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601290 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-token\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601322 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-data\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601375 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601349 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl87z\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-kube-api-access-cl87z\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601379 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d31410-acab-4670-8b2a-1dcacc28a5e8-istiod-ca-cert\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.601456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.601434 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-socket\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.614811 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:21:40.614772 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:21:40.702873 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.702836 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-credential-socket\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.702897 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-token\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.702935 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-data\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.703012 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl87z\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-kube-api-access-cl87z\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.703058 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d31410-acab-4670-8b2a-1dcacc28a5e8-istiod-ca-cert\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703114 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.703091 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-socket\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703408 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.703248 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-certs\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703741 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.703716 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-credential-socket\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703804 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.703759 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-data\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.703958 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.703924 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-envoy\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.704134 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.704022 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-podinfo\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.704523 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.704500 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d31410-acab-4670-8b2a-1dcacc28a5e8-istiod-ca-cert\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.704802 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.704763 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-certs\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.704955 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.704933 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-socket\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.707448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.707373 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-envoy\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.707832 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.707814 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-podinfo\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.757347 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.757314 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-token\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.758105 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.758080 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl87z\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-kube-api-access-cl87z\") pod \"e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.774776 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.774736 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:40.822208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:40.822163 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:41.190877 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:41.190847 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9"] Nov 02 22:21:41.192154 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:21:41.192124 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d31410_acab_4670_8b2a_1dcacc28a5e8.slice/crio-4d70800baf5f6d703c8c90a439d5200abc4a209000d0d675c63450671a0ca9d2 WatchSource:0}: Error finding container 4d70800baf5f6d703c8c90a439d5200abc4a209000d0d675c63450671a0ca9d2: Status 404 returned error can't find the container with id 4d70800baf5f6d703c8c90a439d5200abc4a209000d0d675c63450671a0ca9d2 Nov 02 22:21:41.194310 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:41.194280 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:41.194389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:41.194340 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:41.194389 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:41.194371 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:42.145945 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:42.145900 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" event={"ID":"71d31410-acab-4670-8b2a-1dcacc28a5e8","Type":"ContainerStarted","Data":"14ee22be1d7145b71a9a0a1b60de82e1d02d67f93ed237494b5f22921e169dc5"} Nov 02 22:21:42.146233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:42.145988 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" event={"ID":"71d31410-acab-4670-8b2a-1dcacc28a5e8","Type":"ContainerStarted","Data":"4d70800baf5f6d703c8c90a439d5200abc4a209000d0d675c63450671a0ca9d2"} Nov 02 22:21:42.305899 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:42.305833 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" podStartSLOduration=2.305814099 podStartE2EDuration="2.305814099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:42.254836688 +0000 UTC m=+4131.810465025" watchObservedRunningTime="2026-11-02 22:21:42.305814099 +0000 UTC m=+4131.861442438" Nov 02 22:21:42.775639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:42.775602 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:42.781015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:42.780988 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:43.151961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:43.151931 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:43.152787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:43.152769 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:21:44.996759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:44.996722 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:45.042225 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:45.042183 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr"] Nov 02 22:21:45.089420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:45.089386 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5gqrr"] Nov 02 22:21:45.161246 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:45.161206 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:45.285432 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:45.285349 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:46.582404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:46.582370 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab14f90-1a3a-4736-a8e4-574adc5a0ed5" path="/var/lib/kubelet/pods/aab14f90-1a3a-4736-a8e4-574adc5a0ed5/volumes" Nov 02 22:21:48.219377 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.219343 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:48.341761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.341734 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:48.536009 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.535907 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv"] Nov 02 22:21:48.539800 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.539776 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.592351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.592322 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv"] Nov 02 22:21:48.636840 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.636775 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-89287c1a-dockercfg-74tbs\"" Nov 02 22:21:48.683746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.683714 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c19d19\"" Nov 02 22:21:48.685219 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.685191 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tb8k\" (UniqueName: \"kubernetes.io/projected/d4781700-3187-4061-8cdf-f91462871f10-kube-api-access-6tb8k\") pod \"payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.685361 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.685242 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4781700-3187-4061-8cdf-f91462871f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.693476 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.693451 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr"] Nov 02 22:21:48.697048 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.697030 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:48.749856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.749829 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr"] Nov 02 22:21:48.786831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.786763 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4781700-3187-4061-8cdf-f91462871f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.786831 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.786828 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c76b\" (UniqueName: \"kubernetes.io/projected/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-kube-api-access-4c76b\") pod \"payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:48.787024 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.786858 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:48.787024 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.786903 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tb8k\" (UniqueName: \"kubernetes.io/projected/d4781700-3187-4061-8cdf-f91462871f10-kube-api-access-6tb8k\") pod \"payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.787583 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.787562 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4781700-3187-4061-8cdf-f91462871f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.840588 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.840542 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tb8k\" (UniqueName: \"kubernetes.io/projected/d4781700-3187-4061-8cdf-f91462871f10-kube-api-access-6tb8k\") pod \"payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.853671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.853647 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:48.887842 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.887810 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c76b\" (UniqueName: \"kubernetes.io/projected/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-kube-api-access-4c76b\") pod \"payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:48.887993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.887862 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:48.888575 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.888544 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:48.941519 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:48.941491 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c76b\" (UniqueName: \"kubernetes.io/projected/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-kube-api-access-4c76b\") pod \"payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:49.009720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:49.009695 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:49.219594 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:49.219569 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv"] Nov 02 22:21:49.220423 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:21:49.220392 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4781700_3187_4061_8cdf_f91462871f10.slice/crio-f0c39f6f88ab0c7790ee4cfeaf980fd967fcde7ff8c5b56c0948f759cc847593 WatchSource:0}: Error finding container f0c39f6f88ab0c7790ee4cfeaf980fd967fcde7ff8c5b56c0948f759cc847593: Status 404 returned error can't find the container with id f0c39f6f88ab0c7790ee4cfeaf980fd967fcde7ff8c5b56c0948f759cc847593 Nov 02 22:21:49.376290 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:49.376261 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr"] Nov 02 22:21:49.376765 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:21:49.376739 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e37a3ee_e68e_4ba4_9988_e9c3c088e58a.slice/crio-423bc317249524b520ce14fee7629c7da366c7a69febc1acef5dfe72e699b201 WatchSource:0}: Error finding container 423bc317249524b520ce14fee7629c7da366c7a69febc1acef5dfe72e699b201: Status 404 returned error can't find the container with id 423bc317249524b520ce14fee7629c7da366c7a69febc1acef5dfe72e699b201 Nov 02 22:21:50.195649 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.195599 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" event={"ID":"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a","Type":"ContainerStarted","Data":"d88d2f1eb7523073e2431a5a5b7885940ea4d96ef07362dd6b45a0e4c5e5e84f"} Nov 02 22:21:50.195851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.195761 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:21:50.195851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.195791 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" event={"ID":"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a","Type":"ContainerStarted","Data":"423bc317249524b520ce14fee7629c7da366c7a69febc1acef5dfe72e699b201"} Nov 02 22:21:50.197334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.197298 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" event={"ID":"d4781700-3187-4061-8cdf-f91462871f10","Type":"ContainerStarted","Data":"b7e2274c0dae740d630a936aa0c0da78f6da3c3a90882c73524f0690b0101be5"} Nov 02 22:21:50.197456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.197348 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" event={"ID":"d4781700-3187-4061-8cdf-f91462871f10","Type":"ContainerStarted","Data":"f0c39f6f88ab0c7790ee4cfeaf980fd967fcde7ff8c5b56c0948f759cc847593"} Nov 02 22:21:50.197456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.197368 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:21:50.305575 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.305531 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" podStartSLOduration=2.305517617 podStartE2EDuration="2.305517617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:50.302566839 +0000 UTC m=+4139.858195218" watchObservedRunningTime="2026-11-02 22:21:50.305517617 +0000 UTC m=+4139.861145955" Nov 02 22:21:50.410666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:50.410618 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" podStartSLOduration=2.410605532 podStartE2EDuration="2.410605532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:50.408123065 +0000 UTC m=+4139.963751401" watchObservedRunningTime="2026-11-02 22:21:50.410605532 +0000 UTC m=+4139.966233869" Nov 02 22:21:51.007816 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:51.007758 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:51.158309 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:51.158264 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:51.284395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:51.284318 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:55.644422 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:55.644388 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:56.082125 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:56.082095 4441 scope.go:122] "RemoveContainer" containerID="a4b296f64e4258cb810696eb2d5ca23e81edcaad3466228471c736b704b985e0" Nov 02 22:21:56.091378 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:56.091358 4441 scope.go:122] "RemoveContainer" containerID="f1c68e32778e02f82680ad05559b605e58d032e06af9f7c4c839f1baff29cdf0" Nov 02 22:21:56.099493 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:56.099477 4441 scope.go:122] "RemoveContainer" containerID="b0e6c1ecdfbdb995b9ca90ace5c34f789b6e46640107e677bfd53454fe81002d" Nov 02 22:21:56.110900 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:21:56.110883 4441 scope.go:122] "RemoveContainer" containerID="d2d6bb19e0c9a266996b5b65b64f55ffb5afa333bb6a09430ed760e878f94d3a" Nov 02 22:22:00.286506 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:00.286470 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:01.206996 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:01.206946 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:22:01.207208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:01.207034 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:22:03.658683 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:03.658647 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:04.887080 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:04.887031 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:08.336023 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:08.335962 4441 generic.go:356] "Generic (PLEG): container finished" podID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerID="14ee22be1d7145b71a9a0a1b60de82e1d02d67f93ed237494b5f22921e169dc5" exitCode=0 Nov 02 22:22:08.336568 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:08.336040 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" event={"ID":"71d31410-acab-4670-8b2a-1dcacc28a5e8","Type":"ContainerDied","Data":"14ee22be1d7145b71a9a0a1b60de82e1d02d67f93ed237494b5f22921e169dc5"} Nov 02 22:22:08.336568 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:08.336531 4441 scope.go:122] "RemoveContainer" containerID="14ee22be1d7145b71a9a0a1b60de82e1d02d67f93ed237494b5f22921e169dc5" Nov 02 22:22:08.337810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:08.337782 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:08.337873 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:08.337856 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:08.337911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:08.337891 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:09.343956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:09.343922 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" event={"ID":"71d31410-acab-4670-8b2a-1dcacc28a5e8","Type":"ContainerStarted","Data":"3c9e44c13337958bea8b6681e0fedce78dffb9c89f5c31d05d2dfa5108b9910e"} Nov 02 22:22:09.775456 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:09.775307 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:22:09.789328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:09.783069 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body= Nov 02 22:22:09.789328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:09.783136 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" Nov 02 22:22:09.790351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:09.790280 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:09.914010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:09.913948 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:10.366766 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:10.366735 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:10.607484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:10.607438 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:10.624485 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:22:10.624402 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:22:10.782989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:10.778340 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body= Nov 02 22:22:10.782989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:10.778412 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" Nov 02 22:22:11.777366 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:11.777329 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body= Nov 02 22:22:11.777962 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:11.777931 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" Nov 02 22:22:12.782395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:12.782352 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body= Nov 02 22:22:12.783015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:12.782952 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" Nov 02 22:22:13.152234 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.152198 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:22:13.352196 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.352160 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4"] Nov 02 22:22:13.354895 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.354874 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.395759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.395721 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:13.432181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.432093 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tmp-dir\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.432327 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.432209 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-dshm\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.432327 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.432258 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-model-cache\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.432409 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.432355 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8wq\" (UniqueName: \"kubernetes.io/projected/f75dc399-04eb-492e-b8b3-35c1cc2c3295-kube-api-access-ww8wq\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.432409 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.432379 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-home\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.432494 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.432424 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tls-certs\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.454618 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.454582 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c19d19\"/\"default-dockercfg-fpjwq\"" Nov 02 22:22:13.454827 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.454677 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c19d19\"/\"kube-root-ca.crt\"" Nov 02 22:22:13.454827 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.454779 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c19d19\"/\"simulator-single-main-c19d19-kserve-self-signed-certs\"" Nov 02 22:22:13.456703 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.455440 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c19d19\"/\"openshift-service-ca.crt\"" Nov 02 22:22:13.464016 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.463979 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4"] Nov 02 22:22:13.532894 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.532858 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tls-certs\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.532894 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.532890 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tmp-dir\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.533169 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.532922 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-dshm\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.533169 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.532950 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-model-cache\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.533169 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.533034 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8wq\" (UniqueName: \"kubernetes.io/projected/f75dc399-04eb-492e-b8b3-35c1cc2c3295-kube-api-access-ww8wq\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.533169 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.533063 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-home\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.533412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.533311 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tmp-dir\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.533412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.533385 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-model-cache\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.533499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.533428 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-home\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.535155 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.535136 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-dshm\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.535416 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.535399 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tls-certs\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.590989 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.590287 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8wq\" (UniqueName: \"kubernetes.io/projected/f75dc399-04eb-492e-b8b3-35c1cc2c3295-kube-api-access-ww8wq\") pod \"simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.665562 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.665529 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:13.781992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.780378 4441 patch_prober.go:29] interesting pod/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body= Nov 02 22:22:13.781992 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:13.780442 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" Nov 02 22:22:14.054741 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.054656 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4"] Nov 02 22:22:14.057173 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:22:14.057126 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf75dc399_04eb_492e_b8b3_35c1cc2c3295.slice/crio-9e4c3adb2288fe79486c9665a8b0044877f7c5ff1a00f4b38c673b5d22acae39 WatchSource:0}: Error finding container 9e4c3adb2288fe79486c9665a8b0044877f7c5ff1a00f4b38c673b5d22acae39: Status 404 returned error can't find the container with id 9e4c3adb2288fe79486c9665a8b0044877f7c5ff1a00f4b38c673b5d22acae39 Nov 02 22:22:14.394256 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.394127 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" event={"ID":"f75dc399-04eb-492e-b8b3-35c1cc2c3295","Type":"ContainerStarted","Data":"131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61"} Nov 02 22:22:14.394484 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.394276 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" event={"ID":"f75dc399-04eb-492e-b8b3-35c1cc2c3295","Type":"ContainerStarted","Data":"9e4c3adb2288fe79486c9665a8b0044877f7c5ff1a00f4b38c673b5d22acae39"} Nov 02 22:22:14.475616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.475584 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:14.501691 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.501650 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" podStartSLOduration=1.5016361919999999 podStartE2EDuration="1.501636192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:14.500773372 +0000 UTC m=+4164.056401708" watchObservedRunningTime="2026-11-02 22:22:14.501636192 +0000 UTC m=+4164.057264572" Nov 02 22:22:14.666491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.666302 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:14.779681 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.779652 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:22:14.780650 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:14.780628 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:22:19.046504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:19.046474 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:23.562949 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:23.562912 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:23.666040 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:23.666005 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:23.679081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:23.679056 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:24.464159 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:24.464134 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:22:28.140197 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:28.140163 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:32.637362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:32.637283 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:37.139417 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:37.139384 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:40.614755 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:22:40.614718 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:22:41.667600 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:41.667559 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:46.236821 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:46.236785 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:50.770482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:50.770449 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:51.841127 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:51.841097 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:22:51.848918 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:51.848893 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:22:51.849898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:51.849877 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:22:51.857803 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:51.857781 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:22:51.860252 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:51.860232 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:22:51.868736 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:51.868717 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:22:55.338398 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:55.338363 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:56.144271 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.144238 4441 scope.go:122] "RemoveContainer" containerID="20781a55361d06536d4b342c8b7b3a270939bccb0920430a4d9d7e782442f4a4" Nov 02 22:22:56.154119 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.154095 4441 scope.go:122] "RemoveContainer" containerID="d08706fafcf9366cf2d4c68ee4208886565934b93ee3e90c893d72d0bb33a42e" Nov 02 22:22:56.221029 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.220988 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr"] Nov 02 22:22:56.226486 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.226462 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.255396 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.255368 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.255528 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.255422 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.255528 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.255524 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkccn\" (UniqueName: \"kubernetes.io/projected/63b9fe1c-7799-431e-b400-a563a13997fa-kube-api-access-bkccn\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.278496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.278470 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr"] Nov 02 22:22:56.321322 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.321290 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:22:56.321322 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.321315 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:22:56.321474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.321396 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:22:56.323011 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.322996 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:22:56.356506 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.356482 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkccn\" (UniqueName: \"kubernetes.io/projected/63b9fe1c-7799-431e-b400-a563a13997fa-kube-api-access-bkccn\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.356828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.356551 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.356828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.356599 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.357272 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.357247 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.359015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.358960 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.410080 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.410025 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkccn\" (UniqueName: \"kubernetes.io/projected/63b9fe1c-7799-431e-b400-a563a13997fa-kube-api-access-bkccn\") pod \"maas-api-revoke-sub-3786902c-4dhvr\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.540168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.540143 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:56.902179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:56.901666 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr"] Nov 02 22:22:57.655035 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:57.654999 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" event={"ID":"63b9fe1c-7799-431e-b400-a563a13997fa","Type":"ContainerStarted","Data":"049a164304072595c82beaf9177dfcb9d2754f939fc88f1b8cb2964102e335be"} Nov 02 22:22:57.655421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:57.655060 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" event={"ID":"63b9fe1c-7799-431e-b400-a563a13997fa","Type":"ContainerStarted","Data":"8d0d7020b8bcafb81ebed52d2c96cd02198e2eef2a8251e20deab1e6e3716118"} Nov 02 22:22:57.762233 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:57.762181 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" podStartSLOduration=1.7621662200000001 podStartE2EDuration="1.76216622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:57.761157823 +0000 UTC m=+4207.316786160" watchObservedRunningTime="2026-11-02 22:22:57.76216622 +0000 UTC m=+4207.317794557" Nov 02 22:22:58.662723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:58.662683 4441 generic.go:356] "Generic (PLEG): container finished" podID="63b9fe1c-7799-431e-b400-a563a13997fa" containerID="049a164304072595c82beaf9177dfcb9d2754f939fc88f1b8cb2964102e335be" exitCode=0 Nov 02 22:22:58.663177 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:58.662768 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" event={"ID":"63b9fe1c-7799-431e-b400-a563a13997fa","Type":"ContainerDied","Data":"049a164304072595c82beaf9177dfcb9d2754f939fc88f1b8cb2964102e335be"} Nov 02 22:22:59.821844 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.821818 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:22:59.851238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.851212 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:59.893551 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.893523 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b9fe1c-7799-431e-b400-a563a13997fa-kube-api-access-bkccn\" (UniqueName: \"kubernetes.io/projected/63b9fe1c-7799-431e-b400-a563a13997fa-kube-api-access-bkccn\") pod \"63b9fe1c-7799-431e-b400-a563a13997fa\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " Nov 02 22:22:59.893675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.893575 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-xks-ca\") pod \"63b9fe1c-7799-431e-b400-a563a13997fa\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " Nov 02 22:22:59.893725 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.893672 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-openshift-ca\") pod \"63b9fe1c-7799-431e-b400-a563a13997fa\" (UID: \"63b9fe1c-7799-431e-b400-a563a13997fa\") " Nov 02 22:22:59.894108 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.894078 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-openshift-ca" pod "63b9fe1c-7799-431e-b400-a563a13997fa" (UID: "63b9fe1c-7799-431e-b400-a563a13997fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:59.895646 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.895623 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-xks-ca" pod "63b9fe1c-7799-431e-b400-a563a13997fa" (UID: "63b9fe1c-7799-431e-b400-a563a13997fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:59.895741 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.895647 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b9fe1c-7799-431e-b400-a563a13997fa-kube-api-access-bkccn" pod "63b9fe1c-7799-431e-b400-a563a13997fa" (UID: "63b9fe1c-7799-431e-b400-a563a13997fa"). InnerVolumeSpecName "kube-api-access-bkccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:59.995546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.995463 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkccn\" (UniqueName: \"kubernetes.io/projected/63b9fe1c-7799-431e-b400-a563a13997fa-kube-api-access-bkccn\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:59.995546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.995495 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:59.995546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:22:59.995508 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b9fe1c-7799-431e-b400-a563a13997fa-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:00.675025 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:00.674999 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" Nov 02 22:23:00.675206 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:00.675018 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr" event={"ID":"63b9fe1c-7799-431e-b400-a563a13997fa","Type":"ContainerDied","Data":"8d0d7020b8bcafb81ebed52d2c96cd02198e2eef2a8251e20deab1e6e3716118"} Nov 02 22:23:00.675206 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:00.675066 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0d7020b8bcafb81ebed52d2c96cd02198e2eef2a8251e20deab1e6e3716118" Nov 02 22:23:03.981824 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:03.981787 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:05.419004 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:05.418929 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:07.280352 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.280318 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:07.795909 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.795879 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n"] Nov 02 22:23:07.796300 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.796287 4441 cpu_manager.go:414] "containerMap: removing container" podUID="63b9fe1c-7799-431e-b400-a563a13997fa" containerName="revoke-keys" Nov 02 22:23:07.796362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.796302 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b9fe1c-7799-431e-b400-a563a13997fa" containerName="revoke-keys" Nov 02 22:23:07.796405 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.796383 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b9fe1c-7799-431e-b400-a563a13997fa" containerName="revoke-keys" Nov 02 22:23:07.799563 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.799545 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.852581 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.852540 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n"] Nov 02 22:23:07.868232 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.868208 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9s6\" (UniqueName: \"kubernetes.io/projected/d496c42f-85df-4598-85d0-6be93a58130c-kube-api-access-fh9s6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.868366 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.868242 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.868366 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.868354 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.896461 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.896438 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:23:07.896461 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.896439 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:23:07.896621 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.896446 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:23:07.896896 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.896877 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:23:07.969279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.969250 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9s6\" (UniqueName: \"kubernetes.io/projected/d496c42f-85df-4598-85d0-6be93a58130c-kube-api-access-fh9s6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.969412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.969294 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.969412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.969398 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.970172 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.970150 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:07.972069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:07.972046 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:08.023232 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:08.023204 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9s6\" (UniqueName: \"kubernetes.io/projected/d496c42f-85df-4598-85d0-6be93a58130c-kube-api-access-fh9s6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:08.112444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:08.112415 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:08.481351 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:08.481325 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n"] Nov 02 22:23:08.483242 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:23:08.483209 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd496c42f_85df_4598_85d0_6be93a58130c.slice/crio-99e7b583faf2cd81e82af7819329008bf8c3aadf79899576a09cd7c3bbb74c6b WatchSource:0}: Error finding container 99e7b583faf2cd81e82af7819329008bf8c3aadf79899576a09cd7c3bbb74c6b: Status 404 returned error can't find the container with id 99e7b583faf2cd81e82af7819329008bf8c3aadf79899576a09cd7c3bbb74c6b Nov 02 22:23:08.721785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:08.721748 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" event={"ID":"d496c42f-85df-4598-85d0-6be93a58130c","Type":"ContainerStarted","Data":"99e7b583faf2cd81e82af7819329008bf8c3aadf79899576a09cd7c3bbb74c6b"} Nov 02 22:23:08.991703 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:08.991665 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:09.728885 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:09.728847 4441 generic.go:356] "Generic (PLEG): container finished" podID="d496c42f-85df-4598-85d0-6be93a58130c" containerID="a25a4e354139d25be018eb78312f1291d83fa98417b88e4bab1d58a770bd3de0" exitCode=0 Nov 02 22:23:09.729363 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:09.728937 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" event={"ID":"d496c42f-85df-4598-85d0-6be93a58130c","Type":"ContainerDied","Data":"a25a4e354139d25be018eb78312f1291d83fa98417b88e4bab1d58a770bd3de0"} Nov 02 22:23:10.613710 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:23:10.613674 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:23:10.879444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:10.879386 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:11.001292 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.001261 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-openshift-ca\") pod \"d496c42f-85df-4598-85d0-6be93a58130c\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " Nov 02 22:23:11.001485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.001369 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d496c42f-85df-4598-85d0-6be93a58130c-kube-api-access-fh9s6\" (UniqueName: \"kubernetes.io/projected/d496c42f-85df-4598-85d0-6be93a58130c-kube-api-access-fh9s6\") pod \"d496c42f-85df-4598-85d0-6be93a58130c\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " Nov 02 22:23:11.001485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.001439 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-xks-ca\") pod \"d496c42f-85df-4598-85d0-6be93a58130c\" (UID: \"d496c42f-85df-4598-85d0-6be93a58130c\") " Nov 02 22:23:11.001684 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.001655 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-openshift-ca" pod "d496c42f-85df-4598-85d0-6be93a58130c" (UID: "d496c42f-85df-4598-85d0-6be93a58130c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:11.001808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.001792 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:11.003546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.003518 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-xks-ca" pod "d496c42f-85df-4598-85d0-6be93a58130c" (UID: "d496c42f-85df-4598-85d0-6be93a58130c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:23:11.003664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.003598 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d496c42f-85df-4598-85d0-6be93a58130c-kube-api-access-fh9s6" pod "d496c42f-85df-4598-85d0-6be93a58130c" (UID: "d496c42f-85df-4598-85d0-6be93a58130c"). InnerVolumeSpecName "kube-api-access-fh9s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:11.103266 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.103228 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh9s6\" (UniqueName: \"kubernetes.io/projected/d496c42f-85df-4598-85d0-6be93a58130c-kube-api-access-fh9s6\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:11.103266 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.103260 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d496c42f-85df-4598-85d0-6be93a58130c-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:11.742822 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.742789 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" Nov 02 22:23:11.743061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.742789 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n" event={"ID":"d496c42f-85df-4598-85d0-6be93a58130c","Type":"ContainerDied","Data":"99e7b583faf2cd81e82af7819329008bf8c3aadf79899576a09cd7c3bbb74c6b"} Nov 02 22:23:11.743061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:11.742915 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e7b583faf2cd81e82af7819329008bf8c3aadf79899576a09cd7c3bbb74c6b" Nov 02 22:23:12.246995 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:12.246950 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4"] Nov 02 22:23:12.347843 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:12.347813 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:13.860655 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:13.860611 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:18.352478 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:18.352435 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:19.206242 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:19.206201 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr"] Nov 02 22:23:19.306539 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:19.306499 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv"] Nov 02 22:23:19.541236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:19.541123 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:19.661670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:19.661637 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:22.290771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:22.290737 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:23.722747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:23.722700 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:23.992323 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:23.992247 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:24.206734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:24.206678 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" podUID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" containerName="payload-processing" containerID="cri-o://d88d2f1eb7523073e2431a5a5b7885940ea4d96ef07362dd6b45a0e4c5e5e84f" gracePeriod=25 Nov 02 22:23:24.300760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:24.300680 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n"] Nov 02 22:23:24.307028 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:24.306983 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" podUID="d4781700-3187-4061-8cdf-f91462871f10" containerName="payload-pre-processing" containerID="cri-o://b7e2274c0dae740d630a936aa0c0da78f6da3c3a90882c73524f0690b0101be5" gracePeriod=25 Nov 02 22:23:24.347481 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:24.347441 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c19d19-lm92n"] Nov 02 22:23:24.582798 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:24.582767 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d496c42f-85df-4598-85d0-6be93a58130c" path="/var/lib/kubelet/pods/d496c42f-85df-4598-85d0-6be93a58130c/volumes" Nov 02 22:23:24.705630 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:24.705595 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:27.267192 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.267158 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" podUID="f75dc399-04eb-492e-b8b3-35c1cc2c3295" containerName="main" containerID="cri-o://131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61" gracePeriod=45 Nov 02 22:23:27.519720 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.519666 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:23:27.566028 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.565941 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-home\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-home\") pod \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " Nov 02 22:23:27.566181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566029 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-model-cache\") pod \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " Nov 02 22:23:27.566181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566073 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f75dc399-04eb-492e-b8b3-35c1cc2c3295-kube-api-access-ww8wq\" (UniqueName: \"kubernetes.io/projected/f75dc399-04eb-492e-b8b3-35c1cc2c3295-kube-api-access-ww8wq\") pod \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " Nov 02 22:23:27.566181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566102 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-dshm\") pod \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " Nov 02 22:23:27.566181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566134 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tls-certs\" (UniqueName: \"kubernetes.io/secret/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tls-certs\") pod \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " Nov 02 22:23:27.566181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566159 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tmp-dir\") pod \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\" (UID: \"f75dc399-04eb-492e-b8b3-35c1cc2c3295\") " Nov 02 22:23:27.566181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566157 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-home" pod "f75dc399-04eb-492e-b8b3-35c1cc2c3295" (UID: "f75dc399-04eb-492e-b8b3-35c1cc2c3295"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:27.566543 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566270 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-model-cache" pod "f75dc399-04eb-492e-b8b3-35c1cc2c3295" (UID: "f75dc399-04eb-492e-b8b3-35c1cc2c3295"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:27.566543 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566406 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tmp-dir" pod "f75dc399-04eb-492e-b8b3-35c1cc2c3295" (UID: "f75dc399-04eb-492e-b8b3-35c1cc2c3295"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:27.566670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566627 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:27.566670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566640 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:27.566670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.566648 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:27.568340 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.568318 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tls-certs" pod "f75dc399-04eb-492e-b8b3-35c1cc2c3295" (UID: "f75dc399-04eb-492e-b8b3-35c1cc2c3295"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:23:27.568340 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.568332 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-dshm" pod "f75dc399-04eb-492e-b8b3-35c1cc2c3295" (UID: "f75dc399-04eb-492e-b8b3-35c1cc2c3295"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:27.568482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.568340 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75dc399-04eb-492e-b8b3-35c1cc2c3295-kube-api-access-ww8wq" pod "f75dc399-04eb-492e-b8b3-35c1cc2c3295" (UID: "f75dc399-04eb-492e-b8b3-35c1cc2c3295"). InnerVolumeSpecName "kube-api-access-ww8wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:27.667686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.667643 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww8wq\" (UniqueName: \"kubernetes.io/projected/f75dc399-04eb-492e-b8b3-35c1cc2c3295-kube-api-access-ww8wq\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:27.667686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.667679 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75dc399-04eb-492e-b8b3-35c1cc2c3295-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:27.667686 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.667694 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75dc399-04eb-492e-b8b3-35c1cc2c3295-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:27.838909 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.838875 4441 generic.go:356] "Generic (PLEG): container finished" podID="f75dc399-04eb-492e-b8b3-35c1cc2c3295" containerID="131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61" exitCode=0 Nov 02 22:23:27.839115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.838928 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" Nov 02 22:23:27.839115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.838961 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" event={"ID":"f75dc399-04eb-492e-b8b3-35c1cc2c3295","Type":"ContainerDied","Data":"131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61"} Nov 02 22:23:27.839115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.839032 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" event={"ID":"f75dc399-04eb-492e-b8b3-35c1cc2c3295","Type":"ContainerDied","Data":"9e4c3adb2288fe79486c9665a8b0044877f7c5ff1a00f4b38c673b5d22acae39"} Nov 02 22:23:27.839115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.839057 4441 scope.go:122] "RemoveContainer" containerID="131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61" Nov 02 22:23:27.849372 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.849344 4441 scope.go:122] "RemoveContainer" containerID="131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61" Nov 02 22:23:27.849637 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:23:27.849614 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61\": container with ID starting with 131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61 not found: ID does not exist" containerID="131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61" Nov 02 22:23:27.849722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.849647 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61"} err="failed to get container status \"131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61\": rpc error: code = NotFound desc = could not find container \"131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61\": container with ID starting with 131226af8a453a8e67d2840c0c471bbc37921dc665d1af02d5c878a56c7c6b61 not found: ID does not exist" Nov 02 22:23:27.995670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:27.995632 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4"] Nov 02 22:23:28.041155 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:28.041121 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4"] Nov 02 22:23:28.453346 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:28.453303 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-c19d19/simulator-single-main-c19d19-kserve-6dd5d458b5-74lm4" podUID="f75dc399-04eb-492e-b8b3-35c1cc2c3295" containerName="main" probeResult="failure" output="Get \"https://10.133.1.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:23:28.583161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:28.583128 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f75dc399-04eb-492e-b8b3-35c1cc2c3295" path="/var/lib/kubelet/pods/f75dc399-04eb-492e-b8b3-35c1cc2c3295/volumes" Nov 02 22:23:30.473786 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:30.473753 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:30.610452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:30.610414 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:30.751902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:30.751822 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:30.760645 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:30.760616 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9"] Nov 02 22:23:30.760853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:30.760832 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" containerID="cri-o://3c9e44c13337958bea8b6681e0fedce78dffb9c89f5c31d05d2dfa5108b9910e" gracePeriod=30 Nov 02 22:23:31.042754 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:31.042668 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48"] Nov 02 22:23:31.067756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:31.067719 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319097dc-r4p48"] Nov 02 22:23:31.205216 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:31.205174 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" podUID="d4781700-3187-4061-8cdf-f91462871f10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:23:31.205420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:31.205188 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" podUID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:23:31.254598 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:31.254567 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:31.386181 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:31.386143 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:32.582980 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:32.582935 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b80291d-a49f-405c-9b5e-b7c8d0e48085" path="/var/lib/kubelet/pods/8b80291d-a49f-405c-9b5e-b7c8d0e48085/volumes" Nov 02 22:23:35.891730 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:35.891689 4441 generic.go:356] "Generic (PLEG): container finished" podID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerID="3c9e44c13337958bea8b6681e0fedce78dffb9c89f5c31d05d2dfa5108b9910e" exitCode=0 Nov 02 22:23:35.892184 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:35.891731 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" event={"ID":"71d31410-acab-4670-8b2a-1dcacc28a5e8","Type":"ContainerDied","Data":"3c9e44c13337958bea8b6681e0fedce78dffb9c89f5c31d05d2dfa5108b9910e"} Nov 02 22:23:35.892184 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:35.891846 4441 scope.go:122] "RemoveContainer" containerID="14ee22be1d7145b71a9a0a1b60de82e1d02d67f93ed237494b5f22921e169dc5" Nov 02 22:23:36.009446 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.009422 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:23:36.050272 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050216 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-podinfo\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050272 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050265 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-envoy\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050417 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050292 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-credential-socket\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050417 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050364 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-token\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-token\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050526 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050441 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-certs\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050526 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050478 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d31410-acab-4670-8b2a-1dcacc28a5e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d31410-acab-4670-8b2a-1dcacc28a5e8-istiod-ca-cert\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050648 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050539 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-kube-api-access-cl87z\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-kube-api-access-cl87z\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050648 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050582 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-data\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050645 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-credential-socket" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:36.050761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050662 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-socket\") pod \"71d31410-acab-4670-8b2a-1dcacc28a5e8\" (UID: \"71d31410-acab-4670-8b2a-1dcacc28a5e8\") " Nov 02 22:23:36.050761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.050727 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-certs" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:36.051175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.051101 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.051175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.051131 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.051175 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.051098 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d31410-acab-4670-8b2a-1dcacc28a5e8-istiod-ca-cert" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:36.051424 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.051177 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-socket" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:36.051424 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.051312 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-data" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:36.052566 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.052530 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-podinfo" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:23:36.052685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.052568 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-envoy" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:36.052685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.052605 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-token" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:36.052898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.052881 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-kube-api-access-cl87z" pod "71d31410-acab-4670-8b2a-1dcacc28a5e8" (UID: "71d31410-acab-4670-8b2a-1dcacc28a5e8"). InnerVolumeSpecName "kube-api-access-cl87z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:36.152378 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.152348 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d31410-acab-4670-8b2a-1dcacc28a5e8-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.152378 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.152376 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl87z\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-kube-api-access-cl87z\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.152557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.152393 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.152557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.152402 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.152557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.152411 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.152557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.152419 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.152557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.152427 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d31410-acab-4670-8b2a-1dcacc28a5e8-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:36.901916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.901834 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" Nov 02 22:23:36.901916 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.901876 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9" event={"ID":"71d31410-acab-4670-8b2a-1dcacc28a5e8","Type":"ContainerDied","Data":"4d70800baf5f6d703c8c90a439d5200abc4a209000d0d675c63450671a0ca9d2"} Nov 02 22:23:36.902447 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:36.901945 4441 scope.go:122] "RemoveContainer" containerID="3c9e44c13337958bea8b6681e0fedce78dffb9c89f5c31d05d2dfa5108b9910e" Nov 02 22:23:37.105550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.105513 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9"] Nov 02 22:23:37.108852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.108822 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c19d19-openshift-default-56589d87bf2d6q9"] Nov 02 22:23:37.264893 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.264818 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw"] Nov 02 22:23:37.265235 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265221 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d496c42f-85df-4598-85d0-6be93a58130c" containerName="revoke-keys" Nov 02 22:23:37.265283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265236 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d496c42f-85df-4598-85d0-6be93a58130c" containerName="revoke-keys" Nov 02 22:23:37.265283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265262 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f75dc399-04eb-492e-b8b3-35c1cc2c3295" containerName="main" Nov 02 22:23:37.265283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265267 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f75dc399-04eb-492e-b8b3-35c1cc2c3295" containerName="main" Nov 02 22:23:37.265283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265275 4441 cpu_manager.go:414] "containerMap: removing container" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" Nov 02 22:23:37.265283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265280 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" Nov 02 22:23:37.265454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265287 4441 cpu_manager.go:414] "containerMap: removing container" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" Nov 02 22:23:37.265454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265292 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" Nov 02 22:23:37.265454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265347 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d496c42f-85df-4598-85d0-6be93a58130c" containerName="revoke-keys" Nov 02 22:23:37.265454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265359 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f75dc399-04eb-492e-b8b3-35c1cc2c3295" containerName="main" Nov 02 22:23:37.265454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265365 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" Nov 02 22:23:37.265454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.265371 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" containerName="istio-proxy" Nov 02 22:23:37.269595 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.269576 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.321762 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.320228 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-05df42d0-kserve-self-signed-certs\"" Nov 02 22:23:37.322924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.322894 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw"] Nov 02 22:23:37.363082 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.363058 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:37.365280 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.365255 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-home\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.365344 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.365299 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-tmp-dir\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.365344 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.365324 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-model-cache\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.365460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.365447 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdggl\" (UniqueName: \"kubernetes.io/projected/b57ef793-aa93-4693-9fb1-d069929cd86c-kube-api-access-fdggl\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.365499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.365481 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57ef793-aa93-4693-9fb1-d069929cd86c-tls-certs\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.365614 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.365600 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-dshm\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467019 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.466955 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-dshm\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467046 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-home\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467089 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-tmp-dir\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467111 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-model-cache\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467152 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdggl\" (UniqueName: \"kubernetes.io/projected/b57ef793-aa93-4693-9fb1-d069929cd86c-kube-api-access-fdggl\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467202 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467196 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57ef793-aa93-4693-9fb1-d069929cd86c-tls-certs\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467510 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467487 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-tmp-dir\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467543 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-home\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.467611 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.467546 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-model-cache\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.469395 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.469361 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-dshm\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.469700 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.469678 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57ef793-aa93-4693-9fb1-d069929cd86c-tls-certs\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.521012 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.520950 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdggl\" (UniqueName: \"kubernetes.io/projected/b57ef793-aa93-4693-9fb1-d069929cd86c-kube-api-access-fdggl\") pod \"e2e-same-id-a-05df42d0-kserve-877698565-nvstw\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.581507 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.581478 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:37.948179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:37.948152 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw"] Nov 02 22:23:37.949025 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:23:37.948999 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb57ef793_aa93_4693_9fb1_d069929cd86c.slice/crio-226d48a88aa3f14202769c5098f881dc9d48aa0d11aefd4ed0e97407a510a5bb WatchSource:0}: Error finding container 226d48a88aa3f14202769c5098f881dc9d48aa0d11aefd4ed0e97407a510a5bb: Status 404 returned error can't find the container with id 226d48a88aa3f14202769c5098f881dc9d48aa0d11aefd4ed0e97407a510a5bb Nov 02 22:23:38.583531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:38.583500 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d31410-acab-4670-8b2a-1dcacc28a5e8" path="/var/lib/kubelet/pods/71d31410-acab-4670-8b2a-1dcacc28a5e8/volumes" Nov 02 22:23:38.916564 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:38.916476 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" event={"ID":"b57ef793-aa93-4693-9fb1-d069929cd86c","Type":"ContainerStarted","Data":"6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7"} Nov 02 22:23:38.916564 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:38.916518 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" event={"ID":"b57ef793-aa93-4693-9fb1-d069929cd86c","Type":"ContainerStarted","Data":"226d48a88aa3f14202769c5098f881dc9d48aa0d11aefd4ed0e97407a510a5bb"} Nov 02 22:23:39.027137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:39.027081 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" podStartSLOduration=2.027054131 podStartE2EDuration="2.027054131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:39.024053043 +0000 UTC m=+4248.579681403" watchObservedRunningTime="2026-11-02 22:23:39.027054131 +0000 UTC m=+4248.582682470" Nov 02 22:23:39.582224 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:39.582191 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:40.615360 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:23:40.615324 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:23:41.205739 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:41.205690 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" podUID="d4781700-3187-4061-8cdf-f91462871f10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:23:41.205959 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:41.205731 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" podUID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:23:45.267360 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.266869 4441 generic.go:356] "Generic (PLEG): container finished" podID="d4781700-3187-4061-8cdf-f91462871f10" containerID="b7e2274c0dae740d630a936aa0c0da78f6da3c3a90882c73524f0690b0101be5" exitCode=0 Nov 02 22:23:45.267360 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.267030 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" event={"ID":"d4781700-3187-4061-8cdf-f91462871f10","Type":"ContainerDied","Data":"b7e2274c0dae740d630a936aa0c0da78f6da3c3a90882c73524f0690b0101be5"} Nov 02 22:23:45.306040 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.304911 4441 generic.go:356] "Generic (PLEG): container finished" podID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" containerID="d88d2f1eb7523073e2431a5a5b7885940ea4d96ef07362dd6b45a0e4c5e5e84f" exitCode=0 Nov 02 22:23:45.337145 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.305243 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" event={"ID":"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a","Type":"ContainerDied","Data":"d88d2f1eb7523073e2431a5a5b7885940ea4d96ef07362dd6b45a0e4c5e5e84f"} Nov 02 22:23:45.402993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.402497 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:23:45.411088 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.411064 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:23:45.484118 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484092 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-plugins-config-volume\") pod \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " Nov 02 22:23:45.484289 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484155 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4781700-3187-4061-8cdf-f91462871f10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4781700-3187-4061-8cdf-f91462871f10-plugins-config-volume\") pod \"d4781700-3187-4061-8cdf-f91462871f10\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " Nov 02 22:23:45.484289 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484244 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4781700-3187-4061-8cdf-f91462871f10-kube-api-access-6tb8k\" (UniqueName: \"kubernetes.io/projected/d4781700-3187-4061-8cdf-f91462871f10-kube-api-access-6tb8k\") pod \"d4781700-3187-4061-8cdf-f91462871f10\" (UID: \"d4781700-3187-4061-8cdf-f91462871f10\") " Nov 02 22:23:45.484422 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484311 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-kube-api-access-4c76b\" (UniqueName: \"kubernetes.io/projected/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-kube-api-access-4c76b\") pod \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\" (UID: \"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a\") " Nov 02 22:23:45.484496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484467 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-plugins-config-volume" pod "3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" (UID: "3e37a3ee-e68e-4ba4-9988-e9c3c088e58a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:45.484559 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484532 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4781700-3187-4061-8cdf-f91462871f10-plugins-config-volume" pod "d4781700-3187-4061-8cdf-f91462871f10" (UID: "d4781700-3187-4061-8cdf-f91462871f10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:45.484669 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484653 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:45.484713 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.484675 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4781700-3187-4061-8cdf-f91462871f10-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:45.486376 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.486355 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4781700-3187-4061-8cdf-f91462871f10-kube-api-access-6tb8k" pod "d4781700-3187-4061-8cdf-f91462871f10" (UID: "d4781700-3187-4061-8cdf-f91462871f10"). InnerVolumeSpecName "kube-api-access-6tb8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:45.486445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.486422 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-kube-api-access-4c76b" pod "3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" (UID: "3e37a3ee-e68e-4ba4-9988-e9c3c088e58a"). InnerVolumeSpecName "kube-api-access-4c76b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:45.585734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.585701 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tb8k\" (UniqueName: \"kubernetes.io/projected/d4781700-3187-4061-8cdf-f91462871f10-kube-api-access-6tb8k\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:45.585734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:45.585731 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c76b\" (UniqueName: \"kubernetes.io/projected/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a-kube-api-access-4c76b\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:46.313015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.312962 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" Nov 02 22:23:46.313404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.313013 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr" event={"ID":"3e37a3ee-e68e-4ba4-9988-e9c3c088e58a","Type":"ContainerDied","Data":"423bc317249524b520ce14fee7629c7da366c7a69febc1acef5dfe72e699b201"} Nov 02 22:23:46.313404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.313073 4441 scope.go:122] "RemoveContainer" containerID="d88d2f1eb7523073e2431a5a5b7885940ea4d96ef07362dd6b45a0e4c5e5e84f" Nov 02 22:23:46.314598 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.314568 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" event={"ID":"d4781700-3187-4061-8cdf-f91462871f10","Type":"ContainerDied","Data":"f0c39f6f88ab0c7790ee4cfeaf980fd967fcde7ff8c5b56c0948f759cc847593"} Nov 02 22:23:46.314598 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.314574 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv" Nov 02 22:23:46.323479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.323456 4441 scope.go:122] "RemoveContainer" containerID="b7e2274c0dae740d630a936aa0c0da78f6da3c3a90882c73524f0690b0101be5" Nov 02 22:23:46.468526 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.468470 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv"] Nov 02 22:23:46.471617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.471597 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c19d19-6df95ch8kv"] Nov 02 22:23:46.574262 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.574234 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr"] Nov 02 22:23:46.582744 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.582721 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4781700-3187-4061-8cdf-f91462871f10" path="/var/lib/kubelet/pods/d4781700-3187-4061-8cdf-f91462871f10/volumes" Nov 02 22:23:46.583024 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:46.583011 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c19d19-5bb7bc6d7ptddr"] Nov 02 22:23:47.582092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:47.582055 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:47.594698 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:47.594671 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:48.341309 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:48.341279 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:23:48.582785 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:48.582754 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" path="/var/lib/kubelet/pods/3e37a3ee-e68e-4ba4-9988-e9c3c088e58a/volumes" Nov 02 22:23:48.694879 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:48.694770 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:49.372791 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:49.372756 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:56.180899 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:23:56.180871 4441 scope.go:122] "RemoveContainer" containerID="0abfdc807647f67873364ee49ccf220b7dc96de4be73f536e1ecb4465aadb39d" Nov 02 22:24:03.274918 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:03.274837 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:04.330042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:04.329957 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:10.613653 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:24:10.613618 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:24:18.765368 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.765332 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r"] Nov 02 22:24:18.765758 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.765733 4441 cpu_manager.go:414] "containerMap: removing container" podUID="d4781700-3187-4061-8cdf-f91462871f10" containerName="payload-pre-processing" Nov 02 22:24:18.765758 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.765744 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4781700-3187-4061-8cdf-f91462871f10" containerName="payload-pre-processing" Nov 02 22:24:18.765834 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.765770 4441 cpu_manager.go:414] "containerMap: removing container" podUID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" containerName="payload-processing" Nov 02 22:24:18.765834 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.765775 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" containerName="payload-processing" Nov 02 22:24:18.765910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.765850 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e37a3ee-e68e-4ba4-9988-e9c3c088e58a" containerName="payload-processing" Nov 02 22:24:18.765910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.765864 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4781700-3187-4061-8cdf-f91462871f10" containerName="payload-pre-processing" Nov 02 22:24:18.769095 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.769073 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.849359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.849322 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:18.865936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.865911 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-05df42d0-kserve-self-signed-certs\"" Nov 02 22:24:18.868398 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.868378 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r"] Nov 02 22:24:18.880790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.880766 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tls-certs\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.880882 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.880802 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-home\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.880926 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.880877 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-kube-api-access-mb7xx\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.881012 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.880992 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-dshm\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.881058 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.881028 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-model-cache\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.881095 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.881074 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tmp-dir\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982023 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.981983 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tmp-dir\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982081 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tls-certs\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982121 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-home\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982209 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982172 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-kube-api-access-mb7xx\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982396 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982216 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-dshm\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982396 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982250 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-model-cache\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982514 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982405 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tmp-dir\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982514 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982494 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-home\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.982616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.982575 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-model-cache\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.984412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.984390 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-dshm\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:18.984622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:18.984606 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tls-certs\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:19.040305 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:19.040250 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-kube-api-access-mb7xx\") pod \"e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:19.080135 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:19.080107 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:19.444917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:19.444884 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r"] Nov 02 22:24:19.445820 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:24:19.445790 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fc9e731_1884_4b11_b17a_eb3ab47a2dba.slice/crio-8c55e50b0f785c12fa73443bad5b3e0fca2baa0e88f2d52a4353f749cf316995 WatchSource:0}: Error finding container 8c55e50b0f785c12fa73443bad5b3e0fca2baa0e88f2d52a4353f749cf316995: Status 404 returned error can't find the container with id 8c55e50b0f785c12fa73443bad5b3e0fca2baa0e88f2d52a4353f749cf316995 Nov 02 22:24:19.504454 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:19.504426 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" event={"ID":"5fc9e731-1884-4b11-b17a-eb3ab47a2dba","Type":"ContainerStarted","Data":"20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4"} Nov 02 22:24:19.504592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:19.504468 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" event={"ID":"5fc9e731-1884-4b11-b17a-eb3ab47a2dba","Type":"ContainerStarted","Data":"8c55e50b0f785c12fa73443bad5b3e0fca2baa0e88f2d52a4353f749cf316995"} Nov 02 22:24:20.618391 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:20.618335 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" podStartSLOduration=2.618320347 podStartE2EDuration="2.618320347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:20.618078106 +0000 UTC m=+4290.173706464" watchObservedRunningTime="2026-11-02 22:24:20.618320347 +0000 UTC m=+4290.173948684" Nov 02 22:24:21.081110 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:21.081084 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:26.266123 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:26.266082 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-27kg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:24:26.266781 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:26.266728 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-27kg9" podUID="f47c0164-57f8-4885-a5df-30433184dc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:24:29.080364 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:29.080331 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:29.093139 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:29.093115 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:29.611156 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:29.611116 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:24:40.615425 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:24:40.615385 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:24:52.266697 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:52.266647 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:52.636099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:52.636047 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:24:52.636426 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:52.636380 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:24:56.219119 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:56.219082 4441 scope.go:122] "RemoveContainer" containerID="93e22e881e0f814a3e3b59d511d86bf4b19bc3e7cf13df61b2ac6f2f81c9cb4f" Nov 02 22:24:56.229599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:24:56.229573 4441 scope.go:122] "RemoveContainer" containerID="d37758fd798e9e3f502b48f180066f8f0dc17b194a685235b4a9b83fdcc4ddf1" Nov 02 22:25:01.227243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:01.227211 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw"] Nov 02 22:25:01.261576 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:01.261542 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:04.666617 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:04.666559 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:07.620117 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:07.620073 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hft2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:25:07.620794 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:07.620754 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hft2d" podUID="39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:25:09.682404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:09.682361 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:09.687319 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:09.687296 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r"] Nov 02 22:25:10.614524 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:25:10.614492 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:25:16.248323 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.248255 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" podUID="b57ef793-aa93-4693-9fb1-d069929cd86c" containerName="main" containerID="cri-o://6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7" gracePeriod=45 Nov 02 22:25:16.330479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.330437 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" podUID="b57ef793-aa93-4693-9fb1-d069929cd86c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.74:8000/ready\": dial tcp 10.133.1.74:8000: connect: connection refused" Nov 02 22:25:16.512268 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.512214 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:25:16.679437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679404 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57ef793-aa93-4693-9fb1-d069929cd86c-kube-api-access-fdggl\" (UniqueName: \"kubernetes.io/projected/b57ef793-aa93-4693-9fb1-d069929cd86c-kube-api-access-fdggl\") pod \"b57ef793-aa93-4693-9fb1-d069929cd86c\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " Nov 02 22:25:16.679588 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679448 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-dshm\") pod \"b57ef793-aa93-4693-9fb1-d069929cd86c\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " Nov 02 22:25:16.679588 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679484 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-model-cache\") pod \"b57ef793-aa93-4693-9fb1-d069929cd86c\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " Nov 02 22:25:16.679588 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679534 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-tmp-dir\") pod \"b57ef793-aa93-4693-9fb1-d069929cd86c\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " Nov 02 22:25:16.679747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679594 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57ef793-aa93-4693-9fb1-d069929cd86c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b57ef793-aa93-4693-9fb1-d069929cd86c-tls-certs\") pod \"b57ef793-aa93-4693-9fb1-d069929cd86c\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " Nov 02 22:25:16.679747 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679632 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-home\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-home\") pod \"b57ef793-aa93-4693-9fb1-d069929cd86c\" (UID: \"b57ef793-aa93-4693-9fb1-d069929cd86c\") " Nov 02 22:25:16.679862 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679775 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-model-cache" pod "b57ef793-aa93-4693-9fb1-d069929cd86c" (UID: "b57ef793-aa93-4693-9fb1-d069929cd86c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:16.679926 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679883 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-tmp-dir" pod "b57ef793-aa93-4693-9fb1-d069929cd86c" (UID: "b57ef793-aa93-4693-9fb1-d069929cd86c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:16.680014 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.679917 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-home" pod "b57ef793-aa93-4693-9fb1-d069929cd86c" (UID: "b57ef793-aa93-4693-9fb1-d069929cd86c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:16.680121 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.680100 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.680168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.680129 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.680168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.680144 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.681656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.681626 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-dshm" pod "b57ef793-aa93-4693-9fb1-d069929cd86c" (UID: "b57ef793-aa93-4693-9fb1-d069929cd86c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:16.682252 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.682222 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57ef793-aa93-4693-9fb1-d069929cd86c-tls-certs" pod "b57ef793-aa93-4693-9fb1-d069929cd86c" (UID: "b57ef793-aa93-4693-9fb1-d069929cd86c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:16.682356 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.682251 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57ef793-aa93-4693-9fb1-d069929cd86c-kube-api-access-fdggl" pod "b57ef793-aa93-4693-9fb1-d069929cd86c" (UID: "b57ef793-aa93-4693-9fb1-d069929cd86c"). InnerVolumeSpecName "kube-api-access-fdggl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:16.781086 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.781016 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdggl\" (UniqueName: \"kubernetes.io/projected/b57ef793-aa93-4693-9fb1-d069929cd86c-kube-api-access-fdggl\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.781086 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.781042 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57ef793-aa93-4693-9fb1-d069929cd86c-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.781086 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:16.781052 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57ef793-aa93-4693-9fb1-d069929cd86c-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:17.034326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.034231 4441 generic.go:356] "Generic (PLEG): container finished" podID="b57ef793-aa93-4693-9fb1-d069929cd86c" containerID="6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7" exitCode=0 Nov 02 22:25:17.034326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.034281 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" Nov 02 22:25:17.034326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.034311 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" event={"ID":"b57ef793-aa93-4693-9fb1-d069929cd86c","Type":"ContainerDied","Data":"6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7"} Nov 02 22:25:17.034634 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.034365 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw" event={"ID":"b57ef793-aa93-4693-9fb1-d069929cd86c","Type":"ContainerDied","Data":"226d48a88aa3f14202769c5098f881dc9d48aa0d11aefd4ed0e97407a510a5bb"} Nov 02 22:25:17.034634 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.034387 4441 scope.go:122] "RemoveContainer" containerID="6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7" Nov 02 22:25:17.044489 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.044470 4441 scope.go:122] "RemoveContainer" containerID="6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7" Nov 02 22:25:17.044757 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:25:17.044739 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7\": container with ID starting with 6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7 not found: ID does not exist" containerID="6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7" Nov 02 22:25:17.044827 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.044765 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7"} err="failed to get container status \"6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7\": rpc error: code = NotFound desc = could not find container \"6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7\": container with ID starting with 6c1131f0235cd1cf610938bff020fecca0db4b23da40fafd904703a99d8db6b7 not found: ID does not exist" Nov 02 22:25:17.189383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.189350 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw"] Nov 02 22:25:17.236274 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:17.236244 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-05df42d0-kserve-877698565-nvstw"] Nov 02 22:25:18.582279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:18.582234 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57ef793-aa93-4693-9fb1-d069929cd86c" path="/var/lib/kubelet/pods/b57ef793-aa93-4693-9fb1-d069929cd86c/volumes" Nov 02 22:25:24.706348 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.706294 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" podUID="5fc9e731-1884-4b11-b17a-eb3ab47a2dba" containerName="main" containerID="cri-o://20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4" gracePeriod=45 Nov 02 22:25:24.966418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.966364 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:25:24.991399 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.991375 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-dshm\") pod \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " Nov 02 22:25:24.991524 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.991423 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-model-cache\") pod \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " Nov 02 22:25:24.991524 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.991458 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tmp-dir\") pod \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " Nov 02 22:25:24.991524 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.991484 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-home\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-home\") pod \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " Nov 02 22:25:24.991808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.991779 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-model-cache" pod "5fc9e731-1884-4b11-b17a-eb3ab47a2dba" (UID: "5fc9e731-1884-4b11-b17a-eb3ab47a2dba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:24.991910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.991823 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tmp-dir" pod "5fc9e731-1884-4b11-b17a-eb3ab47a2dba" (UID: "5fc9e731-1884-4b11-b17a-eb3ab47a2dba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:24.991910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.991840 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-home" pod "5fc9e731-1884-4b11-b17a-eb3ab47a2dba" (UID: "5fc9e731-1884-4b11-b17a-eb3ab47a2dba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:24.993412 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:24.993387 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-dshm" pod "5fc9e731-1884-4b11-b17a-eb3ab47a2dba" (UID: "5fc9e731-1884-4b11-b17a-eb3ab47a2dba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:25.092725 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.092693 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-kube-api-access-mb7xx\") pod \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " Nov 02 22:25:25.092900 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.092736 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tls-certs\") pod \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\" (UID: \"5fc9e731-1884-4b11-b17a-eb3ab47a2dba\") " Nov 02 22:25:25.092983 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.092959 4441 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-dshm\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:25.093040 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.092986 4441 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-model-cache\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:25.093040 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.092995 4441 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tmp-dir\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:25.093040 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.093008 4441 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-home\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:25.095095 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.095063 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tls-certs" pod "5fc9e731-1884-4b11-b17a-eb3ab47a2dba" (UID: "5fc9e731-1884-4b11-b17a-eb3ab47a2dba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:25.095232 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.095204 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-kube-api-access-mb7xx" pod "5fc9e731-1884-4b11-b17a-eb3ab47a2dba" (UID: "5fc9e731-1884-4b11-b17a-eb3ab47a2dba"). InnerVolumeSpecName "kube-api-access-mb7xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:25.104429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.104206 4441 generic.go:356] "Generic (PLEG): container finished" podID="5fc9e731-1884-4b11-b17a-eb3ab47a2dba" containerID="20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4" exitCode=0 Nov 02 22:25:25.104429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.104248 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" Nov 02 22:25:25.104429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.104245 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" event={"ID":"5fc9e731-1884-4b11-b17a-eb3ab47a2dba","Type":"ContainerDied","Data":"20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4"} Nov 02 22:25:25.104429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.104367 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r" event={"ID":"5fc9e731-1884-4b11-b17a-eb3ab47a2dba","Type":"ContainerDied","Data":"8c55e50b0f785c12fa73443bad5b3e0fca2baa0e88f2d52a4353f749cf316995"} Nov 02 22:25:25.104429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.104395 4441 scope.go:122] "RemoveContainer" containerID="20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4" Nov 02 22:25:25.114178 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.114159 4441 scope.go:122] "RemoveContainer" containerID="20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4" Nov 02 22:25:25.114460 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:25:25.114442 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4\": container with ID starting with 20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4 not found: ID does not exist" containerID="20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4" Nov 02 22:25:25.114533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.114473 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4"} err="failed to get container status \"20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4\": rpc error: code = NotFound desc = could not find container \"20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4\": container with ID starting with 20e83999a8040eaa555096ef06ab221570356a42dfa17502c002d16992488de4 not found: ID does not exist" Nov 02 22:25:25.193986 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.193944 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-kube-api-access-mb7xx\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:25.193986 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.193984 4441 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc9e731-1884-4b11-b17a-eb3ab47a2dba-tls-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:25.277465 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.277391 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r"] Nov 02 22:25:25.291436 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:25.291409 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-05df42d0-kserve-5975fd68df-q4k7r"] Nov 02 22:25:26.583084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:26.583052 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc9e731-1884-4b11-b17a-eb3ab47a2dba" path="/var/lib/kubelet/pods/5fc9e731-1884-4b11-b17a-eb3ab47a2dba/volumes" Nov 02 22:25:38.074805 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.074766 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7"] Nov 02 22:25:38.075390 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.075368 4441 cpu_manager.go:414] "containerMap: removing container" podUID="5fc9e731-1884-4b11-b17a-eb3ab47a2dba" containerName="main" Nov 02 22:25:38.075498 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.075392 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc9e731-1884-4b11-b17a-eb3ab47a2dba" containerName="main" Nov 02 22:25:38.075498 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.075427 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b57ef793-aa93-4693-9fb1-d069929cd86c" containerName="main" Nov 02 22:25:38.075498 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.075434 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57ef793-aa93-4693-9fb1-d069929cd86c" containerName="main" Nov 02 22:25:38.075664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.075532 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc9e731-1884-4b11-b17a-eb3ab47a2dba" containerName="main" Nov 02 22:25:38.075664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.075550 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57ef793-aa93-4693-9fb1-d069929cd86c" containerName="main" Nov 02 22:25:38.079106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.079082 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.133082 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.132998 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7"] Nov 02 22:25:38.170299 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.170266 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.170479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.170325 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6bvh\" (UniqueName: \"kubernetes.io/projected/77df1d9e-88b7-4985-86b7-3291792f92ac-kube-api-access-d6bvh\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.170479 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.170437 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.176031 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.176009 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:25:38.176168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.176044 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:38.176168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.176049 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:25:38.176660 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.176644 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:25:38.271690 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.271662 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.271852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.271711 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6bvh\" (UniqueName: \"kubernetes.io/projected/77df1d9e-88b7-4985-86b7-3291792f92ac-kube-api-access-d6bvh\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.271852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.271733 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.272378 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.272356 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.274078 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.274053 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.325510 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.325446 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6bvh\" (UniqueName: \"kubernetes.io/projected/77df1d9e-88b7-4985-86b7-3291792f92ac-kube-api-access-d6bvh\") pod \"maas-api-revoke-sub-2077ddf7-flwk7\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.392283 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.392262 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:38.754610 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:38.754582 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7"] Nov 02 22:25:38.755882 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:25:38.755841 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77df1d9e_88b7_4985_86b7_3291792f92ac.slice/crio-16c09acd734b905d12f2d06ca3d63b6a74587986296ad2e19d0e303a047ac9e3 WatchSource:0}: Error finding container 16c09acd734b905d12f2d06ca3d63b6a74587986296ad2e19d0e303a047ac9e3: Status 404 returned error can't find the container with id 16c09acd734b905d12f2d06ca3d63b6a74587986296ad2e19d0e303a047ac9e3 Nov 02 22:25:39.210173 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:39.210133 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" event={"ID":"77df1d9e-88b7-4985-86b7-3291792f92ac","Type":"ContainerStarted","Data":"a679730c0ff9478081f4ace568584684a99d1d86b2b3f8e79908720aadae5c32"} Nov 02 22:25:39.210569 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:39.210198 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" event={"ID":"77df1d9e-88b7-4985-86b7-3291792f92ac","Type":"ContainerStarted","Data":"16c09acd734b905d12f2d06ca3d63b6a74587986296ad2e19d0e303a047ac9e3"} Nov 02 22:25:39.318255 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:39.318203 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" podStartSLOduration=1.3181909090000001 podStartE2EDuration="1.318190909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:39.31693117 +0000 UTC m=+4368.872559525" watchObservedRunningTime="2026-11-02 22:25:39.318190909 +0000 UTC m=+4368.873819245" Nov 02 22:25:40.218533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:40.218498 4441 generic.go:356] "Generic (PLEG): container finished" podID="77df1d9e-88b7-4985-86b7-3291792f92ac" containerID="a679730c0ff9478081f4ace568584684a99d1d86b2b3f8e79908720aadae5c32" exitCode=0 Nov 02 22:25:40.219001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:40.218574 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" event={"ID":"77df1d9e-88b7-4985-86b7-3291792f92ac","Type":"ContainerDied","Data":"a679730c0ff9478081f4ace568584684a99d1d86b2b3f8e79908720aadae5c32"} Nov 02 22:25:40.616405 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:25:40.616358 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:25:41.351810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.351786 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:41.399546 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.399522 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77df1d9e-88b7-4985-86b7-3291792f92ac-kube-api-access-d6bvh\" (UniqueName: \"kubernetes.io/projected/77df1d9e-88b7-4985-86b7-3291792f92ac-kube-api-access-d6bvh\") pod \"77df1d9e-88b7-4985-86b7-3291792f92ac\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " Nov 02 22:25:41.399662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.399565 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-xks-ca\") pod \"77df1d9e-88b7-4985-86b7-3291792f92ac\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " Nov 02 22:25:41.399662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.399598 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-openshift-ca\") pod \"77df1d9e-88b7-4985-86b7-3291792f92ac\" (UID: \"77df1d9e-88b7-4985-86b7-3291792f92ac\") " Nov 02 22:25:41.399979 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.399947 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-openshift-ca" pod "77df1d9e-88b7-4985-86b7-3291792f92ac" (UID: "77df1d9e-88b7-4985-86b7-3291792f92ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:41.401565 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.401535 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77df1d9e-88b7-4985-86b7-3291792f92ac-kube-api-access-d6bvh" pod "77df1d9e-88b7-4985-86b7-3291792f92ac" (UID: "77df1d9e-88b7-4985-86b7-3291792f92ac"). InnerVolumeSpecName "kube-api-access-d6bvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:41.401651 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.401568 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-xks-ca" pod "77df1d9e-88b7-4985-86b7-3291792f92ac" (UID: "77df1d9e-88b7-4985-86b7-3291792f92ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:41.501129 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.501066 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6bvh\" (UniqueName: \"kubernetes.io/projected/77df1d9e-88b7-4985-86b7-3291792f92ac-kube-api-access-d6bvh\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.501129 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.501092 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.501129 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:41.501102 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77df1d9e-88b7-4985-86b7-3291792f92ac-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:42.230935 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:42.230902 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" Nov 02 22:25:42.231156 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:42.230933 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7" event={"ID":"77df1d9e-88b7-4985-86b7-3291792f92ac","Type":"ContainerDied","Data":"16c09acd734b905d12f2d06ca3d63b6a74587986296ad2e19d0e303a047ac9e3"} Nov 02 22:25:42.231156 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:42.230994 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c09acd734b905d12f2d06ca3d63b6a74587986296ad2e19d0e303a047ac9e3" Nov 02 22:25:57.608653 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:57.608601 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:58.120588 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.120545 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm"] Nov 02 22:25:58.120941 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.120929 4441 cpu_manager.go:414] "containerMap: removing container" podUID="77df1d9e-88b7-4985-86b7-3291792f92ac" containerName="revoke-keys" Nov 02 22:25:58.121022 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.120943 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77df1d9e-88b7-4985-86b7-3291792f92ac" containerName="revoke-keys" Nov 02 22:25:58.121064 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.121034 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="77df1d9e-88b7-4985-86b7-3291792f92ac" containerName="revoke-keys" Nov 02 22:25:58.125560 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.125533 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.186507 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.175256 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0abfb3fc-openshift-default-dockercfg-clvvc\"" Nov 02 22:25:58.217361 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.217316 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:58.225602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.225570 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm"] Nov 02 22:25:58.252293 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252260 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-istio-token\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252462 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252310 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5baacd-d754-42bf-8370-fe79e35ab895-istio-podinfo\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252462 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252406 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwd8\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-kube-api-access-mxwd8\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252484 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-envoy\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252543 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-socket\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252649 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-data\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252689 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5baacd-d754-42bf-8370-fe79e35ab895-istiod-ca-cert\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252727 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-credential-socket\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.252830 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.252794 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-certs\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354369 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354328 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-data\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354398 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5baacd-d754-42bf-8370-fe79e35ab895-istiod-ca-cert\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354431 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-credential-socket\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354477 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-certs\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354529 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-istio-token\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354555 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5baacd-d754-42bf-8370-fe79e35ab895-istio-podinfo\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354832 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-data\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.354917 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.354902 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-credential-socket\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.355085 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.355034 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwd8\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-kube-api-access-mxwd8\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.355165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.355089 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-envoy\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.355165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.355148 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-socket\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.355279 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.355204 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-certs\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.355549 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.355528 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-socket\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.355620 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.355550 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5baacd-d754-42bf-8370-fe79e35ab895-istiod-ca-cert\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.357113 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.357093 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5baacd-d754-42bf-8370-fe79e35ab895-istio-podinfo\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.357206 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.357128 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-envoy\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.410571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.410484 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwd8\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-kube-api-access-mxwd8\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.410571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.410522 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-istio-token\") pod \"e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.412780 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.412706 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:58.439089 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.439062 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:58.853069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.853038 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm"] Nov 02 22:25:58.853684 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:25:58.853660 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5baacd_d754_42bf_8370_fe79e35ab895.slice/crio-30fece112db1f82a164cb23015885a69a7d8df34bf8198a80e62c25fc2d4a88d WatchSource:0}: Error finding container 30fece112db1f82a164cb23015885a69a7d8df34bf8198a80e62c25fc2d4a88d: Status 404 returned error can't find the container with id 30fece112db1f82a164cb23015885a69a7d8df34bf8198a80e62c25fc2d4a88d Nov 02 22:25:58.855943 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.855910 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:25:58.856061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.856004 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:25:58.856061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:58.856037 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:25:59.343493 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:59.343456 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" event={"ID":"da5baacd-d754-42bf-8370-fe79e35ab895","Type":"ContainerStarted","Data":"5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9"} Nov 02 22:25:59.343673 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:59.343518 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" event={"ID":"da5baacd-d754-42bf-8370-fe79e35ab895","Type":"ContainerStarted","Data":"30fece112db1f82a164cb23015885a69a7d8df34bf8198a80e62c25fc2d4a88d"} Nov 02 22:25:59.439540 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:59.439505 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:59.444636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:59.444612 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:25:59.454654 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:25:59.454608 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podStartSLOduration=1.454594578 podStartE2EDuration="1.454594578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:59.453676501 +0000 UTC m=+4389.009304849" watchObservedRunningTime="2026-11-02 22:25:59.454594578 +0000 UTC m=+4389.010222914" Nov 02 22:26:00.349295 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:00.349264 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:26:00.350301 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:00.350279 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:26:01.531946 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:01.531915 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:01.696504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:01.696469 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:01.822914 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:01.822878 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:02.050434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:02.050394 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k"] Nov 02 22:26:02.097106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:02.097031 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xh42k"] Nov 02 22:26:02.582756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:02.582721 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2d2367-8f2e-4270-85be-4a6e7690c6fa" path="/var/lib/kubelet/pods/7e2d2367-8f2e-4270-85be-4a6e7690c6fa/volumes" Nov 02 22:26:04.622383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:04.622348 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:04.792191 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:04.792152 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:05.012248 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.012167 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:05.198005 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.197959 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882"] Nov 02 22:26:05.201565 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.201542 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.239268 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.239234 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf6j4\" (UniqueName: \"kubernetes.io/projected/7bf93dbf-7487-4f57-ab19-bffd7fefece2-kube-api-access-gf6j4\") pod \"payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.239418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.239396 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf93dbf-7487-4f57-ab19-bffd7fefece2-plugins-config-volume\") pod \"payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.253828 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.253805 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882"] Nov 02 22:26:05.297676 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.297613 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0abfb3fc\"" Nov 02 22:26:05.305624 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.305603 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0abfb3fc-dockercfg-pcsgj\"" Nov 02 22:26:05.340325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.340297 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf93dbf-7487-4f57-ab19-bffd7fefece2-plugins-config-volume\") pod \"payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.340434 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.340337 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf6j4\" (UniqueName: \"kubernetes.io/projected/7bf93dbf-7487-4f57-ab19-bffd7fefece2-kube-api-access-gf6j4\") pod \"payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.341067 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.341049 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf93dbf-7487-4f57-ab19-bffd7fefece2-plugins-config-volume\") pod \"payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.347637 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.347609 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r"] Nov 02 22:26:05.351171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.351155 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.394499 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.394476 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf6j4\" (UniqueName: \"kubernetes.io/projected/7bf93dbf-7487-4f57-ab19-bffd7fefece2-kube-api-access-gf6j4\") pod \"payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.403759 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.403726 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r"] Nov 02 22:26:05.441853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.441824 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjfp\" (UniqueName: \"kubernetes.io/projected/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-kube-api-access-mzjfp\") pod \"payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.442061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.442040 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.514612 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.514588 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:05.543407 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.543378 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.543541 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.543437 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjfp\" (UniqueName: \"kubernetes.io/projected/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-kube-api-access-mzjfp\") pod \"payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.544390 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.544368 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.597303 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.597270 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjfp\" (UniqueName: \"kubernetes.io/projected/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-kube-api-access-mzjfp\") pod \"payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.663185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.663157 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:05.879124 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:05.879085 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882"] Nov 02 22:26:05.879303 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:26:05.879254 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf93dbf_7487_4f57_ab19_bffd7fefece2.slice/crio-5bbc16ea6b3ae0d6a87a16891305efa4c3d322006db52900b955586fbc5f2736 WatchSource:0}: Error finding container 5bbc16ea6b3ae0d6a87a16891305efa4c3d322006db52900b955586fbc5f2736: Status 404 returned error can't find the container with id 5bbc16ea6b3ae0d6a87a16891305efa4c3d322006db52900b955586fbc5f2736 Nov 02 22:26:06.009141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.009109 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:06.026295 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.026268 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r"] Nov 02 22:26:06.026869 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:26:06.026848 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86b73d7_5dde_4fa1_89ac_65c8f554e97b.slice/crio-055c6a29e92ce3377ba3d5c5bae62f1591c422cee43909fde246842df1f17c01 WatchSource:0}: Error finding container 055c6a29e92ce3377ba3d5c5bae62f1591c422cee43909fde246842df1f17c01: Status 404 returned error can't find the container with id 055c6a29e92ce3377ba3d5c5bae62f1591c422cee43909fde246842df1f17c01 Nov 02 22:26:06.161212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.161143 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:06.385604 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.385570 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" event={"ID":"7bf93dbf-7487-4f57-ab19-bffd7fefece2","Type":"ContainerStarted","Data":"0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708"} Nov 02 22:26:06.385765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.385623 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" event={"ID":"7bf93dbf-7487-4f57-ab19-bffd7fefece2","Type":"ContainerStarted","Data":"5bbc16ea6b3ae0d6a87a16891305efa4c3d322006db52900b955586fbc5f2736"} Nov 02 22:26:06.385765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.385687 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:06.387254 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.387227 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" event={"ID":"b86b73d7-5dde-4fa1-89ac-65c8f554e97b","Type":"ContainerStarted","Data":"f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8"} Nov 02 22:26:06.387369 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.387274 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" event={"ID":"b86b73d7-5dde-4fa1-89ac-65c8f554e97b","Type":"ContainerStarted","Data":"055c6a29e92ce3377ba3d5c5bae62f1591c422cee43909fde246842df1f17c01"} Nov 02 22:26:06.387369 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.387301 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:06.395530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.395507 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:06.497416 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.497334 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" podStartSLOduration=1.4973202589999999 podStartE2EDuration="1.497320259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:06.495078061 +0000 UTC m=+4396.050706411" watchObservedRunningTime="2026-11-02 22:26:06.497320259 +0000 UTC m=+4396.052948595" Nov 02 22:26:06.603045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:06.602982 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" podStartSLOduration=1.6029479210000002 podStartE2EDuration="1.602947921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:06.601329517 +0000 UTC m=+4396.156957883" watchObservedRunningTime="2026-11-02 22:26:06.602947921 +0000 UTC m=+4396.158576259" Nov 02 22:26:10.615032 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:26:10.614997 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:26:10.983023 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:10.982900 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:15.660453 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:15.660416 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:16.715273 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:16.715210 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:17.397013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:17.396953 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:17.397485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:17.397459 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:20.256910 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:20.256831 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:23.492949 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:23.492917 4441 generic.go:356] "Generic (PLEG): container finished" podID="da5baacd-d754-42bf-8370-fe79e35ab895" containerID="5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9" exitCode=0 Nov 02 22:26:23.493429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:23.492998 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" event={"ID":"da5baacd-d754-42bf-8370-fe79e35ab895","Type":"ContainerDied","Data":"5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9"} Nov 02 22:26:23.493429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:23.493363 4441 scope.go:122] "RemoveContainer" containerID="5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9" Nov 02 22:26:23.494506 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:23.494481 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:26:23.494571 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:23.494546 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:26:23.494608 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:23.494577 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:26:24.502797 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:24.502758 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" event={"ID":"da5baacd-d754-42bf-8370-fe79e35ab895","Type":"ContainerStarted","Data":"3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311"} Nov 02 22:26:25.189991 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:25.189549 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:25.440070 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:25.439962 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:26:25.440231 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:25.440189 4441 patch_prober.go:29] interesting pod/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Nov 02 22:26:25.440275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:25.440244 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Nov 02 22:26:26.439913 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:26.439880 4441 patch_prober.go:29] interesting pod/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Nov 02 22:26:26.440373 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:26.439929 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Nov 02 22:26:27.440347 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:27.440313 4441 patch_prober.go:29] interesting pod/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Nov 02 22:26:27.440838 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:27.440367 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Nov 02 22:26:28.439548 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:28.439504 4441 patch_prober.go:29] interesting pod/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Nov 02 22:26:28.439761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:28.439571 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Nov 02 22:26:29.439512 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:29.439464 4441 patch_prober.go:29] interesting pod/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Nov 02 22:26:29.439925 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:29.439536 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Nov 02 22:26:29.780806 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:29.780726 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:30.349445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:30.349410 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:26:30.443717 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:30.443682 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:26:30.539810 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:30.539785 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:26:31.043034 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:31.042996 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd"] Nov 02 22:26:31.045515 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:31.045490 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b6rfd"] Nov 02 22:26:32.582275 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.582244 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39642e40-b045-4031-943e-444bea781ea7" path="/var/lib/kubelet/pods/39642e40-b045-4031-943e-444bea781ea7/volumes" Nov 02 22:26:32.668342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.668311 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6"] Nov 02 22:26:32.671862 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.671843 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.723432 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.723401 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6"] Nov 02 22:26:32.756782 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.756755 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.756938 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.756818 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfp9s\" (UniqueName: \"kubernetes.io/projected/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-kube-api-access-bfp9s\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.757041 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.756990 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.767482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.767457 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:32.767482 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.767472 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:26:32.767663 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.767491 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:26:32.768387 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.768370 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zz6lg\"" Nov 02 22:26:32.858018 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.857914 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.858192 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.858046 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.858192 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.858130 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfp9s\" (UniqueName: \"kubernetes.io/projected/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-kube-api-access-bfp9s\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.858752 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.858726 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.860432 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.860404 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.911863 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.911834 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfp9s\" (UniqueName: \"kubernetes.io/projected/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-kube-api-access-bfp9s\") pod \"maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:32.984320 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:32.984283 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:33.347477 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:33.347443 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6"] Nov 02 22:26:33.348306 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:26:33.348263 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b8a2cc_1cca_4e3b_85f3_871d68bf203e.slice/crio-e77c18ff1111b4150faef8b13bc0fc067fba4019c7b6d2c928140a5a6709c947 WatchSource:0}: Error finding container e77c18ff1111b4150faef8b13bc0fc067fba4019c7b6d2c928140a5a6709c947: Status 404 returned error can't find the container with id e77c18ff1111b4150faef8b13bc0fc067fba4019c7b6d2c928140a5a6709c947 Nov 02 22:26:33.557521 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:33.557488 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" event={"ID":"56b8a2cc-1cca-4e3b-85f3-871d68bf203e","Type":"ContainerStarted","Data":"e77c18ff1111b4150faef8b13bc0fc067fba4019c7b6d2c928140a5a6709c947"} Nov 02 22:26:34.338220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:34.338189 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:34.565496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:34.565459 4441 generic.go:356] "Generic (PLEG): container finished" podID="56b8a2cc-1cca-4e3b-85f3-871d68bf203e" containerID="545468ed6625bd2129af82269f803d6d0eb8ee836b39d3dc261d9ab524e0aa43" exitCode=0 Nov 02 22:26:34.565676 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:34.565544 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" event={"ID":"56b8a2cc-1cca-4e3b-85f3-871d68bf203e","Type":"ContainerDied","Data":"545468ed6625bd2129af82269f803d6d0eb8ee836b39d3dc261d9ab524e0aa43"} Nov 02 22:26:35.712354 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.712331 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:35.784851 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.784821 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-xks-ca\") pod \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " Nov 02 22:26:35.785020 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.784916 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-kube-api-access-bfp9s\" (UniqueName: \"kubernetes.io/projected/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-kube-api-access-bfp9s\") pod \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " Nov 02 22:26:35.785166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.785043 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-openshift-ca\") pod \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\" (UID: \"56b8a2cc-1cca-4e3b-85f3-871d68bf203e\") " Nov 02 22:26:35.785390 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.785369 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-openshift-ca" pod "56b8a2cc-1cca-4e3b-85f3-871d68bf203e" (UID: "56b8a2cc-1cca-4e3b-85f3-871d68bf203e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:35.786852 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.786833 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-kube-api-access-bfp9s" pod "56b8a2cc-1cca-4e3b-85f3-871d68bf203e" (UID: "56b8a2cc-1cca-4e3b-85f3-871d68bf203e"). InnerVolumeSpecName "kube-api-access-bfp9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:35.786905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.786845 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-xks-ca" pod "56b8a2cc-1cca-4e3b-85f3-871d68bf203e" (UID: "56b8a2cc-1cca-4e3b-85f3-871d68bf203e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:35.886656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.886591 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfp9s\" (UniqueName: \"kubernetes.io/projected/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-kube-api-access-bfp9s\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:35.886656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.886620 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:35.886656 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:35.886630 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b8a2cc-1cca-4e3b-85f3-871d68bf203e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.579362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:36.579330 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" Nov 02 22:26:36.583437 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:36.583389 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6" event={"ID":"56b8a2cc-1cca-4e3b-85f3-871d68bf203e","Type":"ContainerDied","Data":"e77c18ff1111b4150faef8b13bc0fc067fba4019c7b6d2c928140a5a6709c947"} Nov 02 22:26:36.583603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:36.583460 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77c18ff1111b4150faef8b13bc0fc067fba4019c7b6d2c928140a5a6709c947" Nov 02 22:26:38.846070 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:38.846037 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:40.615588 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:26:40.615553 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:26:43.344470 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:43.344434 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:44.070790 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:44.070757 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882"] Nov 02 22:26:44.170445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:44.170410 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r"] Nov 02 22:26:44.411366 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:44.411288 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:44.531587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:44.531556 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:47.789708 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:47.789666 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:49.071073 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.071017 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" podUID="7bf93dbf-7487-4f57-ab19-bffd7fefece2" containerName="payload-processing" containerID="cri-o://0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708" gracePeriod=25 Nov 02 22:26:49.171084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.171031 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" podUID="b86b73d7-5dde-4fa1-89ac-65c8f554e97b" containerName="payload-pre-processing" containerID="cri-o://f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8" gracePeriod=25 Nov 02 22:26:49.323164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.323112 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:49.409501 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.409477 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf93dbf-7487-4f57-ab19-bffd7fefece2-kube-api-access-gf6j4\" (UniqueName: \"kubernetes.io/projected/7bf93dbf-7487-4f57-ab19-bffd7fefece2-kube-api-access-gf6j4\") pod \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " Nov 02 22:26:49.409636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.409533 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf93dbf-7487-4f57-ab19-bffd7fefece2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf93dbf-7487-4f57-ab19-bffd7fefece2-plugins-config-volume\") pod \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\" (UID: \"7bf93dbf-7487-4f57-ab19-bffd7fefece2\") " Nov 02 22:26:49.409870 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.409850 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf93dbf-7487-4f57-ab19-bffd7fefece2-plugins-config-volume" pod "7bf93dbf-7487-4f57-ab19-bffd7fefece2" (UID: "7bf93dbf-7487-4f57-ab19-bffd7fefece2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:49.411718 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.411694 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf93dbf-7487-4f57-ab19-bffd7fefece2-kube-api-access-gf6j4" pod "7bf93dbf-7487-4f57-ab19-bffd7fefece2" (UID: "7bf93dbf-7487-4f57-ab19-bffd7fefece2"). InnerVolumeSpecName "kube-api-access-gf6j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:49.414373 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.414355 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:49.510671 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.510643 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-plugins-config-volume\") pod \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " Nov 02 22:26:49.510871 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.510713 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-kube-api-access-mzjfp\" (UniqueName: \"kubernetes.io/projected/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-kube-api-access-mzjfp\") pod \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\" (UID: \"b86b73d7-5dde-4fa1-89ac-65c8f554e97b\") " Nov 02 22:26:49.511044 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.510990 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-plugins-config-volume" pod "b86b73d7-5dde-4fa1-89ac-65c8f554e97b" (UID: "b86b73d7-5dde-4fa1-89ac-65c8f554e97b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:49.511141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.511092 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:49.511141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.511115 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf6j4\" (UniqueName: \"kubernetes.io/projected/7bf93dbf-7487-4f57-ab19-bffd7fefece2-kube-api-access-gf6j4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:49.511141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.511130 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf93dbf-7487-4f57-ab19-bffd7fefece2-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:49.512723 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.512697 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-kube-api-access-mzjfp" pod "b86b73d7-5dde-4fa1-89ac-65c8f554e97b" (UID: "b86b73d7-5dde-4fa1-89ac-65c8f554e97b"). InnerVolumeSpecName "kube-api-access-mzjfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:49.612553 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.612474 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjfp\" (UniqueName: \"kubernetes.io/projected/b86b73d7-5dde-4fa1-89ac-65c8f554e97b-kube-api-access-mzjfp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:49.672746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.672716 4441 generic.go:356] "Generic (PLEG): container finished" podID="7bf93dbf-7487-4f57-ab19-bffd7fefece2" containerID="0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708" exitCode=0 Nov 02 22:26:49.672904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.672761 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" Nov 02 22:26:49.672904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.672795 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" event={"ID":"7bf93dbf-7487-4f57-ab19-bffd7fefece2","Type":"ContainerDied","Data":"0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708"} Nov 02 22:26:49.672904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.672842 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882" event={"ID":"7bf93dbf-7487-4f57-ab19-bffd7fefece2","Type":"ContainerDied","Data":"5bbc16ea6b3ae0d6a87a16891305efa4c3d322006db52900b955586fbc5f2736"} Nov 02 22:26:49.672904 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.672866 4441 scope.go:122] "RemoveContainer" containerID="0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708" Nov 02 22:26:49.674230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.674203 4441 generic.go:356] "Generic (PLEG): container finished" podID="b86b73d7-5dde-4fa1-89ac-65c8f554e97b" containerID="f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8" exitCode=0 Nov 02 22:26:49.674357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.674250 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" Nov 02 22:26:49.674357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.674285 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" event={"ID":"b86b73d7-5dde-4fa1-89ac-65c8f554e97b","Type":"ContainerDied","Data":"f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8"} Nov 02 22:26:49.674357 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.674333 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r" event={"ID":"b86b73d7-5dde-4fa1-89ac-65c8f554e97b","Type":"ContainerDied","Data":"055c6a29e92ce3377ba3d5c5bae62f1591c422cee43909fde246842df1f17c01"} Nov 02 22:26:49.683267 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.683249 4441 scope.go:122] "RemoveContainer" containerID="0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708" Nov 02 22:26:49.683548 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:26:49.683524 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708\": container with ID starting with 0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708 not found: ID does not exist" containerID="0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708" Nov 02 22:26:49.683602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.683558 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708"} err="failed to get container status \"0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708\": rpc error: code = NotFound desc = could not find container \"0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708\": container with ID starting with 0b97448a19f831457b7778c66c768725c6ec7d7e7acad2957605d777ab07d708 not found: ID does not exist" Nov 02 22:26:49.683602 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.683583 4441 scope.go:122] "RemoveContainer" containerID="f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8" Nov 02 22:26:49.692112 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.692095 4441 scope.go:122] "RemoveContainer" containerID="f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8" Nov 02 22:26:49.692377 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:26:49.692358 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8\": container with ID starting with f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8 not found: ID does not exist" containerID="f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8" Nov 02 22:26:49.692426 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.692384 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8"} err="failed to get container status \"f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8\": rpc error: code = NotFound desc = could not find container \"f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8\": container with ID starting with f1e1132c7f6ef2b8f0eeefb98255cac31af8e8fd09b88c9474fa86fee2d352a8 not found: ID does not exist" Nov 02 22:26:49.828392 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.828366 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882"] Nov 02 22:26:49.833763 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.833741 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0abfb3fc-d894d79c-v5882"] Nov 02 22:26:49.936513 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.936437 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r"] Nov 02 22:26:49.939121 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:49.939102 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0abfb3fc-56984c987f-wd88r"] Nov 02 22:26:50.583246 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:50.583216 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf93dbf-7487-4f57-ab19-bffd7fefece2" path="/var/lib/kubelet/pods/7bf93dbf-7487-4f57-ab19-bffd7fefece2/volumes" Nov 02 22:26:50.583641 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:50.583585 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86b73d7-5dde-4fa1-89ac-65c8f554e97b" path="/var/lib/kubelet/pods/b86b73d7-5dde-4fa1-89ac-65c8f554e97b/volumes" Nov 02 22:26:51.687731 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:51.687695 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:53.127196 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:53.127163 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:53.388753 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:53.388679 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:53.597956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:53.597924 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6"] Nov 02 22:26:53.644421 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:53.644347 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0abfb3fc-qxhv6"] Nov 02 22:26:54.582914 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:54.582883 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b8a2cc-1cca-4e3b-85f3-871d68bf203e" path="/var/lib/kubelet/pods/56b8a2cc-1cca-4e3b-85f3-871d68bf203e/volumes" Nov 02 22:26:58.810039 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:58.810005 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm"] Nov 02 22:26:58.810612 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:58.810203 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" containerID="cri-o://3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311" gracePeriod=30 Nov 02 22:26:58.812126 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:58.812098 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:58.930657 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:58.930614 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:59.044381 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:59.044348 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:59.945333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:26:59.945302 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:00.445111 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445075 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9"] Nov 02 22:27:00.445622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445598 4441 cpu_manager.go:414] "containerMap: removing container" podUID="56b8a2cc-1cca-4e3b-85f3-871d68bf203e" containerName="revoke-keys" Nov 02 22:27:00.445622 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445619 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b8a2cc-1cca-4e3b-85f3-871d68bf203e" containerName="revoke-keys" Nov 02 22:27:00.445853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445636 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b86b73d7-5dde-4fa1-89ac-65c8f554e97b" containerName="payload-pre-processing" Nov 02 22:27:00.445853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445643 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86b73d7-5dde-4fa1-89ac-65c8f554e97b" containerName="payload-pre-processing" Nov 02 22:27:00.445853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445684 4441 cpu_manager.go:414] "containerMap: removing container" podUID="7bf93dbf-7487-4f57-ab19-bffd7fefece2" containerName="payload-processing" Nov 02 22:27:00.445853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445692 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf93dbf-7487-4f57-ab19-bffd7fefece2" containerName="payload-processing" Nov 02 22:27:00.445853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445798 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf93dbf-7487-4f57-ab19-bffd7fefece2" containerName="payload-processing" Nov 02 22:27:00.445853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445816 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b8a2cc-1cca-4e3b-85f3-871d68bf203e" containerName="revoke-keys" Nov 02 22:27:00.445853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.445827 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86b73d7-5dde-4fa1-89ac-65c8f554e97b" containerName="payload-pre-processing" Nov 02 22:27:00.450727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.450703 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.501845 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.501814 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-d9f155-openshift-default-dockercfg-slqzl\"" Nov 02 22:27:00.521590 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.521556 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29j2\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-kube-api-access-g29j2\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.521749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.521591 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-envoy\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.521749 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.521721 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0902c12-9ef3-4161-9cba-e15f584407ce-istiod-ca-cert\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.521924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.521754 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-credential-socket\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.521924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.521861 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-token\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.521924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.521884 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-socket\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.522137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.521999 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-data\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.522137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.522023 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-certs\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.522137 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.522043 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-podinfo\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.524382 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.524353 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:00.541548 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.541518 4441 patch_prober.go:29] interesting pod/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:27:00.541679 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.541574 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:27:00.552066 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.552042 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9"] Nov 02 22:27:00.623141 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623112 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29j2\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-kube-api-access-g29j2\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623144 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-envoy\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623203 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0902c12-9ef3-4161-9cba-e15f584407ce-istiod-ca-cert\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623228 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-credential-socket\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623274 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-token\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623304 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-socket\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623345 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-data\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623375 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-certs\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623404 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-podinfo\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623731 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-credential-socket\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623749 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-data\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623829 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-certs\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.623999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.623978 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0902c12-9ef3-4161-9cba-e15f584407ce-istiod-ca-cert\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.624068 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.624046 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-socket\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.625646 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.625625 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-envoy\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.625861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.625842 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-podinfo\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.676881 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.676848 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-token\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.677120 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.677101 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29j2\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-kube-api-access-g29j2\") pod \"e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:00.707042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.697689 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:00.764015 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:00.763961 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:01.176981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.176934 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9"] Nov 02 22:27:01.177731 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:27:01.177703 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0902c12_9ef3_4161_9cba_e15f584407ce.slice/crio-08023829ff320aca5785ec17b809d3023e80a1cce981a8e32a598f5fe571d4a2 WatchSource:0}: Error finding container 08023829ff320aca5785ec17b809d3023e80a1cce981a8e32a598f5fe571d4a2: Status 404 returned error can't find the container with id 08023829ff320aca5785ec17b809d3023e80a1cce981a8e32a598f5fe571d4a2 Nov 02 22:27:01.179765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.179733 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:01.179849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.179798 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:01.179849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.179826 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:01.745031 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.744989 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" event={"ID":"b0902c12-9ef3-4161-9cba-e15f584407ce","Type":"ContainerStarted","Data":"094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004"} Nov 02 22:27:01.745228 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.745057 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" event={"ID":"b0902c12-9ef3-4161-9cba-e15f584407ce","Type":"ContainerStarted","Data":"08023829ff320aca5785ec17b809d3023e80a1cce981a8e32a598f5fe571d4a2"} Nov 02 22:27:01.764276 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.764252 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:01.857756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:01.857701 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" podStartSLOduration=1.857687645 podStartE2EDuration="1.857687645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:01.854872255 +0000 UTC m=+4451.410500591" watchObservedRunningTime="2026-11-02 22:27:01.857687645 +0000 UTC m=+4451.413315981" Nov 02 22:27:02.769107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:02.769076 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:03.756584 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:03.756544 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:03.757493 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:03.757472 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:04.075821 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.075796 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:27:04.158098 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158066 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-data\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158253 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158108 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5baacd-d754-42bf-8370-fe79e35ab895-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5baacd-d754-42bf-8370-fe79e35ab895-istiod-ca-cert\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158253 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158138 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-socket\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158245 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-credential-socket\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158345 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158320 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-istio-token\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-istio-token\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158358 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-certs\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158357 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-data" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:04.158463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158397 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da5baacd-d754-42bf-8370-fe79e35ab895-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5baacd-d754-42bf-8370-fe79e35ab895-istio-podinfo\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158440 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-envoy\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158445 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-socket" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:04.158748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158465 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-kube-api-access-mxwd8\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-kube-api-access-mxwd8\") pod \"da5baacd-d754-42bf-8370-fe79e35ab895\" (UID: \"da5baacd-d754-42bf-8370-fe79e35ab895\") " Nov 02 22:27:04.158748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158515 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-credential-socket" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:04.158748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158569 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5baacd-d754-42bf-8370-fe79e35ab895-istiod-ca-cert" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:04.158748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158739 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-certs" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:04.158999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158851 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.158999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158869 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5baacd-d754-42bf-8370-fe79e35ab895-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.158999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158879 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.158999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158900 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.158999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.158918 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.160643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.160614 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-envoy" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:04.160777 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.160706 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da5baacd-d754-42bf-8370-fe79e35ab895-istio-podinfo" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:04.160777 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.160740 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-kube-api-access-mxwd8" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "kube-api-access-mxwd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:04.160897 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.160881 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-istio-token" pod "da5baacd-d754-42bf-8370-fe79e35ab895" (UID: "da5baacd-d754-42bf-8370-fe79e35ab895"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:04.260166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.260135 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5baacd-d754-42bf-8370-fe79e35ab895-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.260166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.260162 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5baacd-d754-42bf-8370-fe79e35ab895-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.260166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.260171 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwd8\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-kube-api-access-mxwd8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.260397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.260181 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5baacd-d754-42bf-8370-fe79e35ab895-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.483990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.483941 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:04.705153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.705119 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:04.764587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.764494 4441 generic.go:356] "Generic (PLEG): container finished" podID="da5baacd-d754-42bf-8370-fe79e35ab895" containerID="3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311" exitCode=0 Nov 02 22:27:04.764587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.764542 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" Nov 02 22:27:04.764587 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.764580 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" event={"ID":"da5baacd-d754-42bf-8370-fe79e35ab895","Type":"ContainerDied","Data":"3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311"} Nov 02 22:27:04.764809 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.764633 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm" event={"ID":"da5baacd-d754-42bf-8370-fe79e35ab895","Type":"ContainerDied","Data":"30fece112db1f82a164cb23015885a69a7d8df34bf8198a80e62c25fc2d4a88d"} Nov 02 22:27:04.764809 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.764670 4441 scope.go:122] "RemoveContainer" containerID="3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311" Nov 02 22:27:04.773792 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.773775 4441 scope.go:122] "RemoveContainer" containerID="5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9" Nov 02 22:27:04.782101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.782084 4441 scope.go:122] "RemoveContainer" containerID="3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311" Nov 02 22:27:04.782363 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:27:04.782344 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311\": container with ID starting with 3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311 not found: ID does not exist" containerID="3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311" Nov 02 22:27:04.782411 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.782376 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311"} err="failed to get container status \"3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311\": rpc error: code = NotFound desc = could not find container \"3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311\": container with ID starting with 3b316320a247a0e7c5096a43165e44ce4e1f8e9234f610ef27c16bf32a962311 not found: ID does not exist" Nov 02 22:27:04.782411 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.782393 4441 scope.go:122] "RemoveContainer" containerID="5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9" Nov 02 22:27:04.782616 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:27:04.782589 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9\": container with ID starting with 5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9 not found: ID does not exist" containerID="5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9" Nov 02 22:27:04.782717 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.782617 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9"} err="failed to get container status \"5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9\": rpc error: code = NotFound desc = could not find container \"5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9\": container with ID starting with 5d1937f1fd61e72bf9b9fa975e34f0546ab9459dbc32aa3a8a1520914dcdcef9 not found: ID does not exist" Nov 02 22:27:04.829074 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.829051 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:04.972840 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:04.972807 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm"] Nov 02 22:27:05.019935 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:05.019867 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0abfb3fc-openshift-default-6b6598d5c9-wjdtm"] Nov 02 22:27:05.357198 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:05.357164 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:06.582865 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:06.582834 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" path="/var/lib/kubelet/pods/da5baacd-d754-42bf-8370-fe79e35ab895/volumes" Nov 02 22:27:07.640300 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:07.640264 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:07.777226 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:07.777190 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:08.025573 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.025498 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7"] Nov 02 22:27:08.025902 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.025890 4441 cpu_manager.go:414] "containerMap: removing container" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" Nov 02 22:27:08.025947 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.025904 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" Nov 02 22:27:08.025947 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.025921 4441 cpu_manager.go:414] "containerMap: removing container" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" Nov 02 22:27:08.025947 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.025926 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" Nov 02 22:27:08.026099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.026009 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" Nov 02 22:27:08.030521 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.030500 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.081408 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.081375 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7"] Nov 02 22:27:08.125448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.125416 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-d9f155-dockercfg-xgfdm\"" Nov 02 22:27:08.125448 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.125448 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-d9f155\"" Nov 02 22:27:08.180160 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.180126 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj"] Nov 02 22:27:08.180605 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.180591 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5baacd-d754-42bf-8370-fe79e35ab895" containerName="istio-proxy" Nov 02 22:27:08.183787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.183765 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.211287 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.211257 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22637e68-2e92-4794-815c-aec221d8dab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.211287 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.211287 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/22637e68-2e92-4794-815c-aec221d8dab4-kube-api-access-rwq2s\") pod \"payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.237461 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.237435 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj"] Nov 02 22:27:08.311762 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.311734 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22637e68-2e92-4794-815c-aec221d8dab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.311924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.311766 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/22637e68-2e92-4794-815c-aec221d8dab4-kube-api-access-rwq2s\") pod \"payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.311924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.311806 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hjt\" (UniqueName: \"kubernetes.io/projected/64ffb013-5314-4661-90cd-72bfe51625c4-kube-api-access-v6hjt\") pod \"payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.311924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.311865 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffb013-5314-4661-90cd-72bfe51625c4-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.312481 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.312462 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22637e68-2e92-4794-815c-aec221d8dab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.365540 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.365506 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/22637e68-2e92-4794-815c-aec221d8dab4-kube-api-access-rwq2s\") pod \"payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.413036 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.413005 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hjt\" (UniqueName: \"kubernetes.io/projected/64ffb013-5314-4661-90cd-72bfe51625c4-kube-api-access-v6hjt\") pod \"payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.413168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.413079 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffb013-5314-4661-90cd-72bfe51625c4-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.413727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.413706 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffb013-5314-4661-90cd-72bfe51625c4-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.466440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.466415 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hjt\" (UniqueName: \"kubernetes.io/projected/64ffb013-5314-4661-90cd-72bfe51625c4-kube-api-access-v6hjt\") pod \"payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.496719 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.496684 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:08.643208 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.643134 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:08.858987 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:08.858944 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj"] Nov 02 22:27:08.859393 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:27:08.859365 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ffb013_5314_4661_90cd_72bfe51625c4.slice/crio-26aa067221ef8118b83325b6398d4abf616d53b97d52f9b81f908813b579b319 WatchSource:0}: Error finding container 26aa067221ef8118b83325b6398d4abf616d53b97d52f9b81f908813b579b319: Status 404 returned error can't find the container with id 26aa067221ef8118b83325b6398d4abf616d53b97d52f9b81f908813b579b319 Nov 02 22:27:09.006243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.006218 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7"] Nov 02 22:27:09.007256 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:27:09.007234 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22637e68_2e92_4794_815c_aec221d8dab4.slice/crio-51e310b5582b20706a0f957222cc3c0ca45b045d1017021978e75dd435db0fe5 WatchSource:0}: Error finding container 51e310b5582b20706a0f957222cc3c0ca45b045d1017021978e75dd435db0fe5: Status 404 returned error can't find the container with id 51e310b5582b20706a0f957222cc3c0ca45b045d1017021978e75dd435db0fe5 Nov 02 22:27:09.796748 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.796710 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" event={"ID":"64ffb013-5314-4661-90cd-72bfe51625c4","Type":"ContainerStarted","Data":"281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9"} Nov 02 22:27:09.797236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.796770 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" event={"ID":"64ffb013-5314-4661-90cd-72bfe51625c4","Type":"ContainerStarted","Data":"26aa067221ef8118b83325b6398d4abf616d53b97d52f9b81f908813b579b319"} Nov 02 22:27:09.797236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.796863 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:09.798358 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.798332 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" event={"ID":"22637e68-2e92-4794-815c-aec221d8dab4","Type":"ContainerStarted","Data":"6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f"} Nov 02 22:27:09.798480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.798384 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:09.798480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.798400 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" event={"ID":"22637e68-2e92-4794-815c-aec221d8dab4","Type":"ContainerStarted","Data":"51e310b5582b20706a0f957222cc3c0ca45b045d1017021978e75dd435db0fe5"} Nov 02 22:27:09.905786 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:09.905736 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" podStartSLOduration=1.9057232210000001 podStartE2EDuration="1.905723221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:09.903165307 +0000 UTC m=+4459.458793648" watchObservedRunningTime="2026-11-02 22:27:09.905723221 +0000 UTC m=+4459.461351558" Nov 02 22:27:10.011014 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:10.010936 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" podStartSLOduration=3.010919304 podStartE2EDuration="3.010919304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:10.009451391 +0000 UTC m=+4459.565079729" watchObservedRunningTime="2026-11-02 22:27:10.010919304 +0000 UTC m=+4459.566547642" Nov 02 22:27:10.408985 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:10.408943 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:10.558492 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:10.558460 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:10.618167 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:27:10.617402 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:27:10.673288 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:10.673200 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:15.036380 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:15.036339 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:19.651616 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:19.651585 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:20.806866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:20.806837 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:27:20.807335 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:20.806902 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:27:24.251565 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:24.251530 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:26.915634 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:26.915597 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:28.933380 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:28.933344 4441 generic.go:356] "Generic (PLEG): container finished" podID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerID="094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004" exitCode=0 Nov 02 22:27:28.933849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:28.933413 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" event={"ID":"b0902c12-9ef3-4161-9cba-e15f584407ce","Type":"ContainerDied","Data":"094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004"} Nov 02 22:27:28.933849 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:28.933779 4441 scope.go:122] "RemoveContainer" containerID="094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004" Nov 02 22:27:28.934924 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:28.934890 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:28.935042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:28.934961 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:28.935042 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:28.935019 4441 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:29.047583 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:29.047556 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:29.252770 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:29.252667 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:29.942220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:29.942183 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" event={"ID":"b0902c12-9ef3-4161-9cba-e15f584407ce","Type":"ContainerStarted","Data":"ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70"} Nov 02 22:27:30.765159 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:30.765123 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:30.766397 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:30.766370 4441 patch_prober.go:29] interesting pod/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body= Nov 02 22:27:30.766531 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:30.766429 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" Nov 02 22:27:31.764861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:31.764820 4441 patch_prober.go:29] interesting pod/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body= Nov 02 22:27:31.765385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:31.764884 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" Nov 02 22:27:32.764901 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:32.764849 4441 patch_prober.go:29] interesting pod/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body= Nov 02 22:27:32.765420 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:32.764920 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" Nov 02 22:27:33.585993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:33.581278 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:33.757418 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:33.757377 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:33.765226 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:33.765194 4441 patch_prober.go:29] interesting pod/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.765660 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:33.765260 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" Nov 02 22:27:34.764975 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:34.764941 4441 patch_prober.go:29] interesting pod/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body= Nov 02 22:27:34.765168 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:34.765010 4441 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" Nov 02 22:27:35.769212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:35.769183 4441 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:35.770309 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:35.770291 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:27:38.138221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:38.138184 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:40.613881 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:27:40.613842 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:27:42.645238 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:42.645201 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:47.238464 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:47.238427 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:51.846138 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:51.846105 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:51.882309 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:51.882278 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:27:51.890370 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:51.890349 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:27:51.892868 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:51.892850 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:27:51.900480 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:51.900452 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:27:51.901182 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:51.901163 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:27:51.911225 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:51.911208 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:27:56.241815 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:56.241780 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:56.294931 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:56.294889 4441 scope.go:122] "RemoveContainer" containerID="098fa65eff4a5f14c5256b95b614c867e596aed36b4b3ac6b702edac07199605" Nov 02 22:27:56.310670 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:27:56.310646 4441 scope.go:122] "RemoveContainer" containerID="eb2b7921b881fe25960140028dd8c811e8bd0592e19e38cdc34680d9ec71c5e4" Nov 02 22:28:00.043171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:00.043139 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr"] Nov 02 22:28:00.090069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:00.090034 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3786902c-4dhvr"] Nov 02 22:28:00.582801 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:00.582770 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b9fe1c-7799-431e-b400-a563a13997fa" path="/var/lib/kubelet/pods/63b9fe1c-7799-431e-b400-a563a13997fa/volumes" Nov 02 22:28:00.744053 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:00.743982 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:05.334712 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:05.334678 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:06.649011 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:06.648961 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:09.835730 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:09.835696 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:10.614805 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:28:10.614772 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:28:14.495147 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:14.495112 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:18.848540 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:18.848476 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:23.355626 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:23.355588 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:27.841841 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:27.841806 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:32.348775 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:32.348683 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:36.955234 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:36.955193 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:40.614904 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:28:40.614868 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:28:41.338101 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:41.338064 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:45.843880 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:45.843829 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:50.337317 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:50.337277 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:52.197745 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.197706 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8"] Nov 02 22:28:52.201466 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.201443 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.254801 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.254766 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8"] Nov 02 22:28:52.298241 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.298215 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:28:52.298241 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.298246 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:28:52.298440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.298257 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:28:52.298440 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.298253 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zz6lg\"" Nov 02 22:28:52.310765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.310742 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmdg\" (UniqueName: \"kubernetes.io/projected/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-kube-api-access-vqmdg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.310888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.310775 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.310888 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.310806 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.412075 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.412048 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmdg\" (UniqueName: \"kubernetes.io/projected/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-kube-api-access-vqmdg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.412210 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.412083 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.412210 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.412113 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.412826 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.412800 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.414504 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.414483 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.465938 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.465870 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmdg\" (UniqueName: \"kubernetes.io/projected/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-kube-api-access-vqmdg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.514874 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.514850 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:52.881104 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:52.881073 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8"] Nov 02 22:28:52.881995 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:28:52.881944 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64003c8_4a48_4f6e_a50f_3a7d3750ee10.slice/crio-5fdd526869a1da898976709680435fe8a5e6e4839ac5538bead9b57769cb2869 WatchSource:0}: Error finding container 5fdd526869a1da898976709680435fe8a5e6e4839ac5538bead9b57769cb2869: Status 404 returned error can't find the container with id 5fdd526869a1da898976709680435fe8a5e6e4839ac5538bead9b57769cb2869 Nov 02 22:28:53.430452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:53.430371 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" event={"ID":"f64003c8-4a48-4f6e-a50f-3a7d3750ee10","Type":"ContainerStarted","Data":"c71cccda7fc1a50534161985b8dad7a83d26059a2d2c8e1d8b969c0aa5d5bfa7"} Nov 02 22:28:53.430452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:53.430428 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" event={"ID":"f64003c8-4a48-4f6e-a50f-3a7d3750ee10","Type":"ContainerStarted","Data":"5fdd526869a1da898976709680435fe8a5e6e4839ac5538bead9b57769cb2869"} Nov 02 22:28:53.538567 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:53.538519 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" podStartSLOduration=1.538505694 podStartE2EDuration="1.538505694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:28:53.536647583 +0000 UTC m=+4563.092275921" watchObservedRunningTime="2026-11-02 22:28:53.538505694 +0000 UTC m=+4563.094134058" Nov 02 22:28:54.437306 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:54.437269 4441 generic.go:356] "Generic (PLEG): container finished" podID="f64003c8-4a48-4f6e-a50f-3a7d3750ee10" containerID="c71cccda7fc1a50534161985b8dad7a83d26059a2d2c8e1d8b969c0aa5d5bfa7" exitCode=0 Nov 02 22:28:54.437706 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:54.437352 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" event={"ID":"f64003c8-4a48-4f6e-a50f-3a7d3750ee10","Type":"ContainerDied","Data":"c71cccda7fc1a50534161985b8dad7a83d26059a2d2c8e1d8b969c0aa5d5bfa7"} Nov 02 22:28:55.126547 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.126513 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:55.576394 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.576369 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:55.640977 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.640947 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-openshift-ca\") pod \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " Nov 02 22:28:55.641160 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.641106 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-xks-ca\") pod \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " Nov 02 22:28:55.641236 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.641193 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-kube-api-access-vqmdg\" (UniqueName: \"kubernetes.io/projected/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-kube-api-access-vqmdg\") pod \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\" (UID: \"f64003c8-4a48-4f6e-a50f-3a7d3750ee10\") " Nov 02 22:28:55.641371 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.641348 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-openshift-ca" pod "f64003c8-4a48-4f6e-a50f-3a7d3750ee10" (UID: "f64003c8-4a48-4f6e-a50f-3a7d3750ee10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:55.641426 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.641418 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:55.643126 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.643105 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-xks-ca" pod "f64003c8-4a48-4f6e-a50f-3a7d3750ee10" (UID: "f64003c8-4a48-4f6e-a50f-3a7d3750ee10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:55.643216 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.643196 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-kube-api-access-vqmdg" pod "f64003c8-4a48-4f6e-a50f-3a7d3750ee10" (UID: "f64003c8-4a48-4f6e-a50f-3a7d3750ee10"). InnerVolumeSpecName "kube-api-access-vqmdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:55.742682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.742620 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmdg\" (UniqueName: \"kubernetes.io/projected/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-kube-api-access-vqmdg\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:55.742682 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:55.742645 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f64003c8-4a48-4f6e-a50f-3a7d3750ee10-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:56.449337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:56.449303 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" event={"ID":"f64003c8-4a48-4f6e-a50f-3a7d3750ee10","Type":"ContainerDied","Data":"5fdd526869a1da898976709680435fe8a5e6e4839ac5538bead9b57769cb2869"} Nov 02 22:28:56.449337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:56.449334 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8" Nov 02 22:28:56.449554 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:56.449372 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fdd526869a1da898976709680435fe8a5e6e4839ac5538bead9b57769cb2869" Nov 02 22:28:59.947161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:28:59.947130 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:03.650100 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:03.650059 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj"] Nov 02 22:29:03.749619 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:03.749587 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7"] Nov 02 22:29:04.039185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:04.039106 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:04.165583 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:04.165525 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:04.444316 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:04.444280 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:05.650263 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:05.650225 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:05.761949 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:05.761916 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:06.009758 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:06.009677 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:08.651159 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:08.651102 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" podUID="64ffb013-5314-4661-90cd-72bfe51625c4" containerName="payload-processing" containerID="cri-o://281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9" gracePeriod=25 Nov 02 22:29:08.750603 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:08.750533 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" podUID="22637e68-2e92-4794-815c-aec221d8dab4" containerName="payload-pre-processing" containerID="cri-o://6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f" gracePeriod=25 Nov 02 22:29:08.919365 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:08.919298 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:29:08.998407 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:08.998384 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:29:09.070325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070300 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ffb013-5314-4661-90cd-72bfe51625c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffb013-5314-4661-90cd-72bfe51625c4-plugins-config-volume\") pod \"64ffb013-5314-4661-90cd-72bfe51625c4\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " Nov 02 22:29:09.070517 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070364 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22637e68-2e92-4794-815c-aec221d8dab4-kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/22637e68-2e92-4794-815c-aec221d8dab4-kube-api-access-rwq2s\") pod \"22637e68-2e92-4794-815c-aec221d8dab4\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " Nov 02 22:29:09.070517 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070396 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22637e68-2e92-4794-815c-aec221d8dab4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22637e68-2e92-4794-815c-aec221d8dab4-plugins-config-volume\") pod \"22637e68-2e92-4794-815c-aec221d8dab4\" (UID: \"22637e68-2e92-4794-815c-aec221d8dab4\") " Nov 02 22:29:09.070517 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070464 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffb013-5314-4661-90cd-72bfe51625c4-kube-api-access-v6hjt\" (UniqueName: \"kubernetes.io/projected/64ffb013-5314-4661-90cd-72bfe51625c4-kube-api-access-v6hjt\") pod \"64ffb013-5314-4661-90cd-72bfe51625c4\" (UID: \"64ffb013-5314-4661-90cd-72bfe51625c4\") " Nov 02 22:29:09.070760 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070728 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ffb013-5314-4661-90cd-72bfe51625c4-plugins-config-volume" pod "64ffb013-5314-4661-90cd-72bfe51625c4" (UID: "64ffb013-5314-4661-90cd-72bfe51625c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:09.070808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070752 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22637e68-2e92-4794-815c-aec221d8dab4-plugins-config-volume" pod "22637e68-2e92-4794-815c-aec221d8dab4" (UID: "22637e68-2e92-4794-815c-aec221d8dab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:09.070850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070821 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22637e68-2e92-4794-815c-aec221d8dab4-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:09.070850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.070837 4441 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ffb013-5314-4661-90cd-72bfe51625c4-plugins-config-volume\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:09.072390 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.072368 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22637e68-2e92-4794-815c-aec221d8dab4-kube-api-access-rwq2s" pod "22637e68-2e92-4794-815c-aec221d8dab4" (UID: "22637e68-2e92-4794-815c-aec221d8dab4"). InnerVolumeSpecName "kube-api-access-rwq2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:09.072490 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.072467 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffb013-5314-4661-90cd-72bfe51625c4-kube-api-access-v6hjt" pod "64ffb013-5314-4661-90cd-72bfe51625c4" (UID: "64ffb013-5314-4661-90cd-72bfe51625c4"). InnerVolumeSpecName "kube-api-access-v6hjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:09.171485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.171408 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/22637e68-2e92-4794-815c-aec221d8dab4-kube-api-access-rwq2s\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:09.171485 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.171439 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6hjt\" (UniqueName: \"kubernetes.io/projected/64ffb013-5314-4661-90cd-72bfe51625c4-kube-api-access-v6hjt\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:09.523128 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.523040 4441 generic.go:356] "Generic (PLEG): container finished" podID="64ffb013-5314-4661-90cd-72bfe51625c4" containerID="281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9" exitCode=0 Nov 02 22:29:09.523128 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.523085 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" Nov 02 22:29:09.523342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.523121 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" event={"ID":"64ffb013-5314-4661-90cd-72bfe51625c4","Type":"ContainerDied","Data":"281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9"} Nov 02 22:29:09.523342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.523168 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj" event={"ID":"64ffb013-5314-4661-90cd-72bfe51625c4","Type":"ContainerDied","Data":"26aa067221ef8118b83325b6398d4abf616d53b97d52f9b81f908813b579b319"} Nov 02 22:29:09.523342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.523190 4441 scope.go:122] "RemoveContainer" containerID="281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9" Nov 02 22:29:09.524623 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.524600 4441 generic.go:356] "Generic (PLEG): container finished" podID="22637e68-2e92-4794-815c-aec221d8dab4" containerID="6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f" exitCode=0 Nov 02 22:29:09.524751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.524655 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" event={"ID":"22637e68-2e92-4794-815c-aec221d8dab4","Type":"ContainerDied","Data":"6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f"} Nov 02 22:29:09.524751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.524687 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" event={"ID":"22637e68-2e92-4794-815c-aec221d8dab4","Type":"ContainerDied","Data":"51e310b5582b20706a0f957222cc3c0ca45b045d1017021978e75dd435db0fe5"} Nov 02 22:29:09.524751 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.524654 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7" Nov 02 22:29:09.533326 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.533310 4441 scope.go:122] "RemoveContainer" containerID="281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9" Nov 02 22:29:09.533595 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:29:09.533571 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9\": container with ID starting with 281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9 not found: ID does not exist" containerID="281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9" Nov 02 22:29:09.533703 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.533602 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9"} err="failed to get container status \"281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9\": rpc error: code = NotFound desc = could not find container \"281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9\": container with ID starting with 281d3a1c60da1393eac9b8d899d8d797131ec639549762e20bd943d6e63a11d9 not found: ID does not exist" Nov 02 22:29:09.533703 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.533620 4441 scope.go:122] "RemoveContainer" containerID="6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f" Nov 02 22:29:09.541664 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.541643 4441 scope.go:122] "RemoveContainer" containerID="6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f" Nov 02 22:29:09.541909 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:29:09.541891 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f\": container with ID starting with 6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f not found: ID does not exist" containerID="6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f" Nov 02 22:29:09.541960 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.541915 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f"} err="failed to get container status \"6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f\": rpc error: code = NotFound desc = could not find container \"6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f\": container with ID starting with 6ede760ccab5e7264e4cd01e2d8ce45515ee9b23d9aea0d2b4b592aa62d0ce6f not found: ID does not exist" Nov 02 22:29:09.678385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.678358 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7"] Nov 02 22:29:09.725278 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.725247 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d9f155-595bf59c4c-7vgc7"] Nov 02 22:29:09.827788 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.827766 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj"] Nov 02 22:29:09.861919 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:09.861893 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d9f155-57d66cf5bc-bh2tj"] Nov 02 22:29:10.581743 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:10.581710 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22637e68-2e92-4794-815c-aec221d8dab4" path="/var/lib/kubelet/pods/22637e68-2e92-4794-815c-aec221d8dab4/volumes" Nov 02 22:29:10.582134 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:10.582119 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ffb013-5314-4661-90cd-72bfe51625c4" path="/var/lib/kubelet/pods/64ffb013-5314-4661-90cd-72bfe51625c4/volumes" Nov 02 22:29:10.614035 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:29:10.613999 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:29:13.454473 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:13.454439 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:14.873264 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:14.873225 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:15.134833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:15.134753 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:15.463885 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:15.463804 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8"] Nov 02 22:29:15.512918 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:15.512886 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d9f155-7mpr8"] Nov 02 22:29:15.861415 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:15.861383 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:16.582337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:16.582305 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f64003c8-4a48-4f6e-a50f-3a7d3750ee10" path="/var/lib/kubelet/pods/f64003c8-4a48-4f6e-a50f-3a7d3750ee10/volumes" Nov 02 22:29:21.690333 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:21.690255 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:21.821325 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:21.821289 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:21.978013 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:21.977914 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:21.982309 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:21.982287 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9"] Nov 02 22:29:21.982501 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:21.982469 4441 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" containerID="cri-o://ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70" gracePeriod=30 Nov 02 22:29:24.410012 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:24.409938 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:27.525999 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.524805 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:29:27.570383 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.570343 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-data\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.571601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.570624 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-socket\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.572142 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.572119 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-socket" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:27.572445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.572426 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-token\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-token\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.572636 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.572618 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-certs\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.573539 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.573519 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-podinfo\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.573804 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.573786 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0902c12-9ef3-4161-9cba-e15f584407ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0902c12-9ef3-4161-9cba-e15f584407ce-istiod-ca-cert\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.574068 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.574041 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-envoy\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.574836 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.574815 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-credential-socket\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.575030 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.575004 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-kube-api-access-g29j2\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-kube-api-access-g29j2\") pod \"b0902c12-9ef3-4161-9cba-e15f584407ce\" (UID: \"b0902c12-9ef3-4161-9cba-e15f584407ce\") " Nov 02 22:29:27.575577 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.575557 4441 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.576239 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.573078 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-certs" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:27.576385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.573454 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-data" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:27.576820 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.576796 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0902c12-9ef3-4161-9cba-e15f584407ce-istiod-ca-cert" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:27.587987 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.585060 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-credential-socket" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:27.593103 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.593067 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-envoy" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:27.593328 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.593308 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-kube-api-access-g29j2" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "kube-api-access-g29j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:27.611151 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.611107 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-podinfo" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:29:27.613474 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.611515 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-token" pod "b0902c12-9ef3-4161-9cba-e15f584407ce" (UID: "b0902c12-9ef3-4161-9cba-e15f584407ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:27.653213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.651589 4441 generic.go:356] "Generic (PLEG): container finished" podID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerID="ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70" exitCode=0 Nov 02 22:29:27.653213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.651647 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" event={"ID":"b0902c12-9ef3-4161-9cba-e15f584407ce","Type":"ContainerDied","Data":"ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70"} Nov 02 22:29:27.653213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.651695 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" event={"ID":"b0902c12-9ef3-4161-9cba-e15f584407ce","Type":"ContainerDied","Data":"08023829ff320aca5785ec17b809d3023e80a1cce981a8e32a598f5fe571d4a2"} Nov 02 22:29:27.653213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.651723 4441 scope.go:122] "RemoveContainer" containerID="ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70" Nov 02 22:29:27.653213 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.651894 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9" Nov 02 22:29:27.681337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.681261 4441 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-token\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.681586 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.681525 4441 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-workload-certs\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.681586 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.681550 4441 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-podinfo\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.681854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.681565 4441 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0902c12-9ef3-4161-9cba-e15f584407ce-istiod-ca-cert\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.681854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.681798 4441 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-envoy\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.681854 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.681813 4441 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-credential-socket\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.682148 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.681828 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29j2\" (UniqueName: \"kubernetes.io/projected/b0902c12-9ef3-4161-9cba-e15f584407ce-kube-api-access-g29j2\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.682148 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.682126 4441 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0902c12-9ef3-4161-9cba-e15f584407ce-istio-data\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:27.791382 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:27.791352 4441 scope.go:122] "RemoveContainer" containerID="094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004" Nov 02 22:29:28.021471 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:28.020837 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9"] Nov 02 22:29:28.059067 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:28.052667 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d9f155-openshift-default-7c8fbc7f6f-tbrw9"] Nov 02 22:29:28.099001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:28.088166 4441 scope.go:122] "RemoveContainer" containerID="ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70" Nov 02 22:29:28.099001 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:29:28.088620 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70\": container with ID starting with ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70 not found: ID does not exist" containerID="ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70" Nov 02 22:29:28.099001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:28.088664 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70"} err="failed to get container status \"ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70\": rpc error: code = NotFound desc = could not find container \"ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70\": container with ID starting with ea4bf21b6b5ddac2cda1e59de3837f0e6b78ccd6a01b8ccb3dc6ae4442c86e70 not found: ID does not exist" Nov 02 22:29:28.099001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:28.088694 4441 scope.go:122] "RemoveContainer" containerID="094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004" Nov 02 22:29:28.099001 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:29:28.089014 4441 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004\": container with ID starting with 094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004 not found: ID does not exist" containerID="094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004" Nov 02 22:29:28.099001 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:28.089042 4441 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004"} err="failed to get container status \"094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004\": rpc error: code = NotFound desc = could not find container \"094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004\": container with ID starting with 094a829c48de518ded7f68d24efb8e603a9071e1991571ec4184cf9860010004 not found: ID does not exist" Nov 02 22:29:28.610002 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:28.606024 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" path="/var/lib/kubelet/pods/b0902c12-9ef3-4161-9cba-e15f584407ce/volumes" Nov 02 22:29:40.246491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:40.246447 4441 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-js4dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:40.247611 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:40.247576 4441 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-js4dr" podUID="be8ea2f8-52aa-4e10-bca6-b01563cd940f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:40.716358 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:29:40.716312 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:29:45.730765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.730729 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f"] Nov 02 22:29:45.731164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731110 4441 cpu_manager.go:414] "containerMap: removing container" podUID="f64003c8-4a48-4f6e-a50f-3a7d3750ee10" containerName="revoke-keys" Nov 02 22:29:45.731164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731121 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64003c8-4a48-4f6e-a50f-3a7d3750ee10" containerName="revoke-keys" Nov 02 22:29:45.731164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731144 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" Nov 02 22:29:45.731164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731149 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" Nov 02 22:29:45.731164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731158 4441 cpu_manager.go:414] "containerMap: removing container" podUID="22637e68-2e92-4794-815c-aec221d8dab4" containerName="payload-pre-processing" Nov 02 22:29:45.731164 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731164 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22637e68-2e92-4794-815c-aec221d8dab4" containerName="payload-pre-processing" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731175 4441 cpu_manager.go:414] "containerMap: removing container" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731180 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731191 4441 cpu_manager.go:414] "containerMap: removing container" podUID="64ffb013-5314-4661-90cd-72bfe51625c4" containerName="payload-processing" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731196 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ffb013-5314-4661-90cd-72bfe51625c4" containerName="payload-processing" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731254 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="22637e68-2e92-4794-815c-aec221d8dab4" containerName="payload-pre-processing" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731264 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731271 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ffb013-5314-4661-90cd-72bfe51625c4" containerName="payload-processing" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731278 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64003c8-4a48-4f6e-a50f-3a7d3750ee10" containerName="revoke-keys" Nov 02 22:29:45.731385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.731286 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0902c12-9ef3-4161-9cba-e15f584407ce" containerName="istio-proxy" Nov 02 22:29:45.734359 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.734342 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.787407 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.787381 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f"] Nov 02 22:29:45.831179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.831148 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:45.831179 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.831158 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:29:45.831358 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.831178 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zz6lg\"" Nov 02 22:29:45.831358 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.831179 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:29:45.844342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.844315 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.844481 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.844361 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.844554 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.844539 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fknq5\" (UniqueName: \"kubernetes.io/projected/ecf8ce91-f038-4559-969c-4e29fa65004e-kube-api-access-fknq5\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.945430 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.945396 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fknq5\" (UniqueName: \"kubernetes.io/projected/ecf8ce91-f038-4559-969c-4e29fa65004e-kube-api-access-fknq5\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.945579 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.945471 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.945579 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.945516 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.946210 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.946188 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.947734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.947717 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:45.999334 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:45.999269 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fknq5\" (UniqueName: \"kubernetes.io/projected/ecf8ce91-f038-4559-969c-4e29fa65004e-kube-api-access-fknq5\") pod \"maas-api-revoke-sub-f03629db-dr87f\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:46.046478 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:46.046452 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:46.408726 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:46.408699 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f"] Nov 02 22:29:46.409903 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:29:46.409878 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf8ce91_f038_4559_969c_4e29fa65004e.slice/crio-28be239397ba525527f906abe1c24b04e36dcb10a9928b226190809862095cca WatchSource:0}: Error finding container 28be239397ba525527f906abe1c24b04e36dcb10a9928b226190809862095cca: Status 404 returned error can't find the container with id 28be239397ba525527f906abe1c24b04e36dcb10a9928b226190809862095cca Nov 02 22:29:46.823165 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:46.823129 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" event={"ID":"ecf8ce91-f038-4559-969c-4e29fa65004e","Type":"ContainerStarted","Data":"35d93bd7a52cac0fffc13d6a35fa08f87cdf2c887a899e4026c0f9dfbea41630"} Nov 02 22:29:46.823530 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:46.823192 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" event={"ID":"ecf8ce91-f038-4559-969c-4e29fa65004e","Type":"ContainerStarted","Data":"28be239397ba525527f906abe1c24b04e36dcb10a9928b226190809862095cca"} Nov 02 22:29:46.933171 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:46.932921 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" podStartSLOduration=1.932903355 podStartE2EDuration="1.932903355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:46.929358557 +0000 UTC m=+4616.484986894" watchObservedRunningTime="2026-11-02 22:29:46.932903355 +0000 UTC m=+4616.488531693" Nov 02 22:29:47.829870 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:47.829831 4441 generic.go:356] "Generic (PLEG): container finished" podID="ecf8ce91-f038-4559-969c-4e29fa65004e" containerID="35d93bd7a52cac0fffc13d6a35fa08f87cdf2c887a899e4026c0f9dfbea41630" exitCode=0 Nov 02 22:29:47.830300 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:47.829916 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" event={"ID":"ecf8ce91-f038-4559-969c-4e29fa65004e","Type":"ContainerDied","Data":"35d93bd7a52cac0fffc13d6a35fa08f87cdf2c887a899e4026c0f9dfbea41630"} Nov 02 22:29:48.972476 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:48.972453 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:49.076463 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.076435 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-xks-ca\") pod \"ecf8ce91-f038-4559-969c-4e29fa65004e\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " Nov 02 22:29:49.076639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.076484 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-openshift-ca\") pod \"ecf8ce91-f038-4559-969c-4e29fa65004e\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " Nov 02 22:29:49.076639 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.076540 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf8ce91-f038-4559-969c-4e29fa65004e-kube-api-access-fknq5\" (UniqueName: \"kubernetes.io/projected/ecf8ce91-f038-4559-969c-4e29fa65004e-kube-api-access-fknq5\") pod \"ecf8ce91-f038-4559-969c-4e29fa65004e\" (UID: \"ecf8ce91-f038-4559-969c-4e29fa65004e\") " Nov 02 22:29:49.076887 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.076864 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-openshift-ca" pod "ecf8ce91-f038-4559-969c-4e29fa65004e" (UID: "ecf8ce91-f038-4559-969c-4e29fa65004e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:49.078445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.078417 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf8ce91-f038-4559-969c-4e29fa65004e-kube-api-access-fknq5" pod "ecf8ce91-f038-4559-969c-4e29fa65004e" (UID: "ecf8ce91-f038-4559-969c-4e29fa65004e"). InnerVolumeSpecName "kube-api-access-fknq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:49.078445 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.078437 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-xks-ca" pod "ecf8ce91-f038-4559-969c-4e29fa65004e" (UID: "ecf8ce91-f038-4559-969c-4e29fa65004e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:49.178161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.178092 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:49.178161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.178119 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf8ce91-f038-4559-969c-4e29fa65004e-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:49.178161 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.178130 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fknq5\" (UniqueName: \"kubernetes.io/projected/ecf8ce91-f038-4559-969c-4e29fa65004e-kube-api-access-fknq5\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:49.842156 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.842128 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" Nov 02 22:29:49.842337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.842122 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dr87f" event={"ID":"ecf8ce91-f038-4559-969c-4e29fa65004e","Type":"ContainerDied","Data":"28be239397ba525527f906abe1c24b04e36dcb10a9928b226190809862095cca"} Nov 02 22:29:49.842337 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:49.842258 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28be239397ba525527f906abe1c24b04e36dcb10a9928b226190809862095cca" Nov 02 22:29:56.357732 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:56.357676 4441 scope.go:122] "RemoveContainer" containerID="049a164304072595c82beaf9177dfcb9d2754f939fc88f1b8cb2964102e335be" Nov 02 22:29:56.367401 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:29:56.367379 4441 scope.go:122] "RemoveContainer" containerID="a25a4e354139d25be018eb78312f1291d83fa98417b88e4bab1d58a770bd3de0" Nov 02 22:30:00.160727 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.160694 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b"] Nov 02 22:30:00.161154 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.161093 4441 cpu_manager.go:414] "containerMap: removing container" podUID="ecf8ce91-f038-4559-969c-4e29fa65004e" containerName="revoke-keys" Nov 02 22:30:00.161154 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.161104 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf8ce91-f038-4559-969c-4e29fa65004e" containerName="revoke-keys" Nov 02 22:30:00.161234 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.161180 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf8ce91-f038-4559-969c-4e29fa65004e" containerName="revoke-keys" Nov 02 22:30:00.165898 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.165877 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.216593 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.216556 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b"] Nov 02 22:30:00.260833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.260802 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:30:00.260833 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.260817 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:30:00.262452 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.262435 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zz6lg\"" Nov 02 22:30:00.282178 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.282152 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzng4\" (UniqueName: \"kubernetes.io/projected/0094b188-7f7c-4cf2-81dc-4817a3ae5772-kube-api-access-dzng4\") pod \"maas-api-key-cleanup-29894310-qx99b\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.282332 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.282188 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094b188-7f7c-4cf2-81dc-4817a3ae5772-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-qx99b\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.383468 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.383437 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzng4\" (UniqueName: \"kubernetes.io/projected/0094b188-7f7c-4cf2-81dc-4817a3ae5772-kube-api-access-dzng4\") pod \"maas-api-key-cleanup-29894310-qx99b\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.383652 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.383475 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094b188-7f7c-4cf2-81dc-4817a3ae5772-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-qx99b\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.384113 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.384093 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094b188-7f7c-4cf2-81dc-4817a3ae5772-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-qx99b\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.437232 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.437157 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzng4\" (UniqueName: \"kubernetes.io/projected/0094b188-7f7c-4cf2-81dc-4817a3ae5772-kube-api-access-dzng4\") pod \"maas-api-key-cleanup-29894310-qx99b\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.478365 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.478336 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:00.843308 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.843277 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b"] Nov 02 22:30:00.843440 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:30:00.843314 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0094b188_7f7c_4cf2_81dc_4817a3ae5772.slice/crio-defa120e87f0fa874c99fc572c3ccb62728c32a370316fa0a9d4f405b8e632dc WatchSource:0}: Error finding container defa120e87f0fa874c99fc572c3ccb62728c32a370316fa0a9d4f405b8e632dc: Status 404 returned error can't find the container with id defa120e87f0fa874c99fc572c3ccb62728c32a370316fa0a9d4f405b8e632dc Nov 02 22:30:00.914362 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:00.914326 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" event={"ID":"0094b188-7f7c-4cf2-81dc-4817a3ae5772","Type":"ContainerStarted","Data":"defa120e87f0fa874c99fc572c3ccb62728c32a370316fa0a9d4f405b8e632dc"} Nov 02 22:30:01.921981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:01.921863 4441 generic.go:356] "Generic (PLEG): container finished" podID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerID="219ede6a6ddbaa911e2b2f3dda1fc6f062bf4f0405a1255193bb54bc69218b38" exitCode=60 Nov 02 22:30:01.921981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:01.921944 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" event={"ID":"0094b188-7f7c-4cf2-81dc-4817a3ae5772","Type":"ContainerDied","Data":"219ede6a6ddbaa911e2b2f3dda1fc6f062bf4f0405a1255193bb54bc69218b38"} Nov 02 22:30:01.922435 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:01.922233 4441 scope.go:122] "RemoveContainer" containerID="219ede6a6ddbaa911e2b2f3dda1fc6f062bf4f0405a1255193bb54bc69218b38" Nov 02 22:30:02.929756 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:02.929675 4441 generic.go:356] "Generic (PLEG): container finished" podID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerID="6cee2e524e143cbb30c659ac845235baaad182479b8bd0931ee697153411d825" exitCode=60 Nov 02 22:30:02.930166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:02.929762 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" event={"ID":"0094b188-7f7c-4cf2-81dc-4817a3ae5772","Type":"ContainerDied","Data":"6cee2e524e143cbb30c659ac845235baaad182479b8bd0931ee697153411d825"} Nov 02 22:30:02.930166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:02.929828 4441 scope.go:122] "RemoveContainer" containerID="219ede6a6ddbaa911e2b2f3dda1fc6f062bf4f0405a1255193bb54bc69218b38" Nov 02 22:30:02.930166 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:02.930061 4441 scope.go:122] "RemoveContainer" containerID="6cee2e524e143cbb30c659ac845235baaad182479b8bd0931ee697153411d825" Nov 02 22:30:02.930335 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:30:02.930310 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-qx99b_odh-ai-gateway-infra(0094b188-7f7c-4cf2-81dc-4817a3ae5772)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" Nov 02 22:30:03.945491 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:03.944017 4441 scope.go:122] "RemoveContainer" containerID="6cee2e524e143cbb30c659ac845235baaad182479b8bd0931ee697153411d825" Nov 02 22:30:03.945491 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:30:03.944308 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-qx99b_odh-ai-gateway-infra(0094b188-7f7c-4cf2-81dc-4817a3ae5772)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" Nov 02 22:30:06.366626 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.366590 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:06.520478 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.520443 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6"] Nov 02 22:30:06.525249 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.525228 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.576534 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.576508 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6"] Nov 02 22:30:06.606891 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.606856 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.607092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.606992 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.607092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.607063 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwdp\" (UniqueName: \"kubernetes.io/projected/e78e8c53-0934-47ee-9565-c50c1912757d-kube-api-access-vqwdp\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.619863 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.619809 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:30:06.708228 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.708189 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.708422 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.708277 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwdp\" (UniqueName: \"kubernetes.io/projected/e78e8c53-0934-47ee-9565-c50c1912757d-kube-api-access-vqwdp\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.708422 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.708324 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.709065 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.709045 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.710567 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.710544 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.762212 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.762173 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwdp\" (UniqueName: \"kubernetes.io/projected/e78e8c53-0934-47ee-9565-c50c1912757d-kube-api-access-vqwdp\") pod \"maas-api-revoke-sub-0165d7d2-tsmw6\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:06.837866 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:06.837837 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:07.201592 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:07.201560 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6"] Nov 02 22:30:07.202940 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:30:07.202063 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78e8c53_0934_47ee_9565_c50c1912757d.slice/crio-16412c85bbc2ff189ad3f86402b9d0e867d794d678a9db7f927d9f45900f9821 WatchSource:0}: Error finding container 16412c85bbc2ff189ad3f86402b9d0e867d794d678a9db7f927d9f45900f9821: Status 404 returned error can't find the container with id 16412c85bbc2ff189ad3f86402b9d0e867d794d678a9db7f927d9f45900f9821 Nov 02 22:30:07.990377 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:07.990340 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" event={"ID":"e78e8c53-0934-47ee-9565-c50c1912757d","Type":"ContainerStarted","Data":"5190adc3dd6ab12b19f517e6dbd8a0fc0941cd9a76d5b2f89456836796bbde0d"} Nov 02 22:30:07.990765 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:07.990391 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" event={"ID":"e78e8c53-0934-47ee-9565-c50c1912757d","Type":"ContainerStarted","Data":"16412c85bbc2ff189ad3f86402b9d0e867d794d678a9db7f927d9f45900f9821"} Nov 02 22:30:08.099091 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:08.099039 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" podStartSLOduration=2.099023741 podStartE2EDuration="2.099023741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:08.096095359 +0000 UTC m=+4637.651723699" watchObservedRunningTime="2026-11-02 22:30:08.099023741 +0000 UTC m=+4637.654652077" Nov 02 22:30:08.997398 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:08.997362 4441 generic.go:356] "Generic (PLEG): container finished" podID="e78e8c53-0934-47ee-9565-c50c1912757d" containerID="5190adc3dd6ab12b19f517e6dbd8a0fc0941cd9a76d5b2f89456836796bbde0d" exitCode=0 Nov 02 22:30:08.997779 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:08.997444 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" event={"ID":"e78e8c53-0934-47ee-9565-c50c1912757d","Type":"ContainerDied","Data":"5190adc3dd6ab12b19f517e6dbd8a0fc0941cd9a76d5b2f89456836796bbde0d"} Nov 02 22:30:10.136823 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.136799 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:10.240034 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.240000 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-openshift-ca\") pod \"e78e8c53-0934-47ee-9565-c50c1912757d\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " Nov 02 22:30:10.240228 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.240074 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78e8c53-0934-47ee-9565-c50c1912757d-kube-api-access-vqwdp\" (UniqueName: \"kubernetes.io/projected/e78e8c53-0934-47ee-9565-c50c1912757d-kube-api-access-vqwdp\") pod \"e78e8c53-0934-47ee-9565-c50c1912757d\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " Nov 02 22:30:10.240228 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.240124 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-xks-ca\") pod \"e78e8c53-0934-47ee-9565-c50c1912757d\" (UID: \"e78e8c53-0934-47ee-9565-c50c1912757d\") " Nov 02 22:30:10.240409 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.240385 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-openshift-ca" pod "e78e8c53-0934-47ee-9565-c50c1912757d" (UID: "e78e8c53-0934-47ee-9565-c50c1912757d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:10.242128 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.242095 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-xks-ca" pod "e78e8c53-0934-47ee-9565-c50c1912757d" (UID: "e78e8c53-0934-47ee-9565-c50c1912757d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:10.242245 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.242156 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78e8c53-0934-47ee-9565-c50c1912757d-kube-api-access-vqwdp" pod "e78e8c53-0934-47ee-9565-c50c1912757d" (UID: "e78e8c53-0934-47ee-9565-c50c1912757d"). InnerVolumeSpecName "kube-api-access-vqwdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:10.341092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.341063 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqwdp\" (UniqueName: \"kubernetes.io/projected/e78e8c53-0934-47ee-9565-c50c1912757d-kube-api-access-vqwdp\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:10.341092 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.341090 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:10.341296 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:10.341101 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78e8c53-0934-47ee-9565-c50c1912757d-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:10.613879 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:30:10.613809 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:30:11.009651 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:11.009576 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" event={"ID":"e78e8c53-0934-47ee-9565-c50c1912757d","Type":"ContainerDied","Data":"16412c85bbc2ff189ad3f86402b9d0e867d794d678a9db7f927d9f45900f9821"} Nov 02 22:30:11.009651 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:11.009595 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tsmw6" Nov 02 22:30:11.009651 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:11.009622 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16412c85bbc2ff189ad3f86402b9d0e867d794d678a9db7f927d9f45900f9821" Nov 02 22:30:15.577053 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:15.577025 4441 scope.go:122] "RemoveContainer" containerID="6cee2e524e143cbb30c659ac845235baaad182479b8bd0931ee697153411d825" Nov 02 22:30:16.037069 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:16.037033 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" event={"ID":"0094b188-7f7c-4cf2-81dc-4817a3ae5772","Type":"ContainerStarted","Data":"edd3f41c638d2dcfdf80f5eaf12c3f3def728fbdcfa2574aba32eec682a1b647"} Nov 02 22:30:16.144666 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:16.144576 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" podStartSLOduration=16.144555106 podStartE2EDuration="16.144555106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:16.143825937 +0000 UTC m=+4645.699454275" watchObservedRunningTime="2026-11-02 22:30:16.144555106 +0000 UTC m=+4645.700183442" Nov 02 22:30:17.044450 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:17.044414 4441 generic.go:356] "Generic (PLEG): container finished" podID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerID="edd3f41c638d2dcfdf80f5eaf12c3f3def728fbdcfa2574aba32eec682a1b647" exitCode=60 Nov 02 22:30:17.044936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:17.044491 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" event={"ID":"0094b188-7f7c-4cf2-81dc-4817a3ae5772","Type":"ContainerDied","Data":"edd3f41c638d2dcfdf80f5eaf12c3f3def728fbdcfa2574aba32eec682a1b647"} Nov 02 22:30:17.044936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:17.044571 4441 scope.go:122] "RemoveContainer" containerID="6cee2e524e143cbb30c659ac845235baaad182479b8bd0931ee697153411d825" Nov 02 22:30:17.044936 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:17.044861 4441 scope.go:122] "RemoveContainer" containerID="edd3f41c638d2dcfdf80f5eaf12c3f3def728fbdcfa2574aba32eec682a1b647" Nov 02 22:30:17.045164 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:30:17.045140 4441 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-qx99b_odh-ai-gateway-infra(0094b188-7f7c-4cf2-81dc-4817a3ae5772)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" Nov 02 22:30:17.154856 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:17.154829 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b"] Nov 02 22:30:18.187107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.187073 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57"] Nov 02 22:30:18.187681 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.187657 4441 cpu_manager.go:414] "containerMap: removing container" podUID="e78e8c53-0934-47ee-9565-c50c1912757d" containerName="revoke-keys" Nov 02 22:30:18.187771 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.187682 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78e8c53-0934-47ee-9565-c50c1912757d" containerName="revoke-keys" Nov 02 22:30:18.187829 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.187797 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78e8c53-0934-47ee-9565-c50c1912757d" containerName="revoke-keys" Nov 02 22:30:18.192331 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.192310 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.193784 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.193762 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:18.241526 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.241501 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:30:18.243054 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.243031 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57"] Nov 02 22:30:18.306404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.306340 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0094b188-7f7c-4cf2-81dc-4817a3ae5772-kube-api-access-dzng4\" (UniqueName: \"kubernetes.io/projected/0094b188-7f7c-4cf2-81dc-4817a3ae5772-kube-api-access-dzng4\") pod \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " Nov 02 22:30:18.306522 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.306430 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0094b188-7f7c-4cf2-81dc-4817a3ae5772-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094b188-7f7c-4cf2-81dc-4817a3ae5772-maas-api-service-ca\") pod \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\" (UID: \"0094b188-7f7c-4cf2-81dc-4817a3ae5772\") " Nov 02 22:30:18.306685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.306663 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.306738 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.306722 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.306781 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.306762 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0094b188-7f7c-4cf2-81dc-4817a3ae5772-maas-api-service-ca" pod "0094b188-7f7c-4cf2-81dc-4817a3ae5772" (UID: "0094b188-7f7c-4cf2-81dc-4817a3ae5772"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:18.306860 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.306842 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkh8\" (UniqueName: \"kubernetes.io/projected/5149b207-d098-403e-8c57-ed43a9d27406-kube-api-access-lfkh8\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.306957 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.306945 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094b188-7f7c-4cf2-81dc-4817a3ae5772-maas-api-service-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:18.308502 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.308484 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0094b188-7f7c-4cf2-81dc-4817a3ae5772-kube-api-access-dzng4" pod "0094b188-7f7c-4cf2-81dc-4817a3ae5772" (UID: "0094b188-7f7c-4cf2-81dc-4817a3ae5772"). InnerVolumeSpecName "kube-api-access-dzng4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:18.408195 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.407486 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkh8\" (UniqueName: \"kubernetes.io/projected/5149b207-d098-403e-8c57-ed43a9d27406-kube-api-access-lfkh8\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.408195 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.407629 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.408195 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.407673 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.408195 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.407742 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzng4\" (UniqueName: \"kubernetes.io/projected/0094b188-7f7c-4cf2-81dc-4817a3ae5772-kube-api-access-dzng4\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:18.409280 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.409248 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.410889 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.410844 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.462294 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.462260 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkh8\" (UniqueName: \"kubernetes.io/projected/5149b207-d098-403e-8c57-ed43a9d27406-kube-api-access-lfkh8\") pod \"maas-api-revoke-sub-6b19338f-frg57\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.504255 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.504227 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:18.867398 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:18.867369 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57"] Nov 02 22:30:18.868862 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:30:18.868830 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5149b207_d098_403e_8c57_ed43a9d27406.slice/crio-06759c4f300d5ded74a385451cac5ed5942a8743a77fdbecd253f693b9dc5467 WatchSource:0}: Error finding container 06759c4f300d5ded74a385451cac5ed5942a8743a77fdbecd253f693b9dc5467: Status 404 returned error can't find the container with id 06759c4f300d5ded74a385451cac5ed5942a8743a77fdbecd253f693b9dc5467 Nov 02 22:30:19.060084 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:19.060051 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" Nov 02 22:30:19.060247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:19.060058 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b" event={"ID":"0094b188-7f7c-4cf2-81dc-4817a3ae5772","Type":"ContainerDied","Data":"defa120e87f0fa874c99fc572c3ccb62728c32a370316fa0a9d4f405b8e632dc"} Nov 02 22:30:19.060247 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:19.060183 4441 scope.go:122] "RemoveContainer" containerID="edd3f41c638d2dcfdf80f5eaf12c3f3def728fbdcfa2574aba32eec682a1b647" Nov 02 22:30:19.061507 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:19.061431 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" event={"ID":"5149b207-d098-403e-8c57-ed43a9d27406","Type":"ContainerStarted","Data":"06759c4f300d5ded74a385451cac5ed5942a8743a77fdbecd253f693b9dc5467"} Nov 02 22:30:19.217008 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:19.216928 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b"] Nov 02 22:30:19.220643 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:19.220618 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-qx99b"] Nov 02 22:30:20.069533 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:20.069500 4441 generic.go:356] "Generic (PLEG): container finished" podID="5149b207-d098-403e-8c57-ed43a9d27406" containerID="7348c16e4d42a3694f081eed6d92e49083eedfff9edc39ff1401fca3fa1045ff" exitCode=0 Nov 02 22:30:20.069733 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:20.069592 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" event={"ID":"5149b207-d098-403e-8c57-ed43a9d27406","Type":"ContainerDied","Data":"7348c16e4d42a3694f081eed6d92e49083eedfff9edc39ff1401fca3fa1045ff"} Nov 02 22:30:20.582710 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:20.582681 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" path="/var/lib/kubelet/pods/0094b188-7f7c-4cf2-81dc-4817a3ae5772/volumes" Nov 02 22:30:21.214861 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.214833 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:21.332151 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.332120 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5149b207-d098-403e-8c57-ed43a9d27406-kube-api-access-lfkh8\" (UniqueName: \"kubernetes.io/projected/5149b207-d098-403e-8c57-ed43a9d27406-kube-api-access-lfkh8\") pod \"5149b207-d098-403e-8c57-ed43a9d27406\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " Nov 02 22:30:21.332318 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.332179 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-xks-ca\") pod \"5149b207-d098-403e-8c57-ed43a9d27406\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " Nov 02 22:30:21.332318 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.332203 4441 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-openshift-ca\") pod \"5149b207-d098-403e-8c57-ed43a9d27406\" (UID: \"5149b207-d098-403e-8c57-ed43a9d27406\") " Nov 02 22:30:21.332679 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.332650 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-openshift-ca" pod "5149b207-d098-403e-8c57-ed43a9d27406" (UID: "5149b207-d098-403e-8c57-ed43a9d27406"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:21.334961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.334900 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-xks-ca" pod "5149b207-d098-403e-8c57-ed43a9d27406" (UID: "5149b207-d098-403e-8c57-ed43a9d27406"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:21.334961 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.334939 4441 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5149b207-d098-403e-8c57-ed43a9d27406-kube-api-access-lfkh8" pod "5149b207-d098-403e-8c57-ed43a9d27406" (UID: "5149b207-d098-403e-8c57-ed43a9d27406"). InnerVolumeSpecName "kube-api-access-lfkh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:21.433353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.433324 4441 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfkh8\" (UniqueName: \"kubernetes.io/projected/5149b207-d098-403e-8c57-ed43a9d27406-kube-api-access-lfkh8\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:21.433353 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.433351 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-xks-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:21.433518 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:21.433360 4441 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5149b207-d098-403e-8c57-ed43a9d27406-maas-api-openshift-ca\") on node \"ip-10-0-137-225.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.090115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:22.090079 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" event={"ID":"5149b207-d098-403e-8c57-ed43a9d27406","Type":"ContainerDied","Data":"06759c4f300d5ded74a385451cac5ed5942a8743a77fdbecd253f693b9dc5467"} Nov 02 22:30:22.090115 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:22.090107 4441 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-frg57" Nov 02 22:30:22.090599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:22.090130 4441 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06759c4f300d5ded74a385451cac5ed5942a8743a77fdbecd253f693b9dc5467" Nov 02 22:30:30.313227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:30.313190 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c57f8d5-mq6lj_33243403-0984-43ce-90f3-04befaf56f63/authorino/0.log" Nov 02 22:30:40.565850 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:40.565817 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bc68ffc7c-pmmp6_636400c6-9f4c-476d-964b-2bf5a279c586/manager/0.log" Nov 02 22:30:40.614445 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:30:40.614406 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:30:42.047385 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:42.047350 4441 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7"] Nov 02 22:30:42.065249 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:42.065224 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9f8dffcbc-95qc8_f0776fe1-d7f2-4f43-b0e0-a9495263c354/manager/0.log" Nov 02 22:30:42.094573 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:42.094545 4441 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2077ddf7-flwk7"] Nov 02 22:30:42.589331 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:42.583825 4441 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77df1d9e-88b7-4985-86b7-3291792f92ac" path="/var/lib/kubelet/pods/77df1d9e-88b7-4985-86b7-3291792f92ac/volumes" Nov 02 22:30:44.443028 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:44.442997 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-tsmw6_e78e8c53-0934-47ee-9565-c50c1912757d/revoke-keys/0.log" Nov 02 22:30:44.670444 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:44.670415 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-frg57_5149b207-d098-403e-8c57-ed43a9d27406/revoke-keys/0.log" Nov 02 22:30:44.888941 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:44.888912 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dr87f_ecf8ce91-f038-4559-969c-4e29fa65004e/revoke-keys/0.log" Nov 02 22:30:46.551174 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:46.551145 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c57f8d5-mq6lj_33243403-0984-43ce-90f3-04befaf56f63/authorino/0.log" Nov 02 22:30:47.011823 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:47.011793 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-g8lw8_73d816d8-936a-4ff8-a6e6-f49cb87788a5/manager/0.log" Nov 02 22:30:47.245717 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:47.245688 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-4zzv5_2bff1a62-9aa4-478c-a9a9-f95e1c1fd094/manager/0.log" Nov 02 22:30:47.454892 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:47.454864 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6d9v2_e000b5aa-e884-4263-8bc7-1b64c302bd10/kuadrant-console-plugin/0.log" Nov 02 22:30:47.690339 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:47.690301 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5qm9c_6298a9da-6f99-4b43-93d0-7082cd03311b/manager/0.log" Nov 02 22:30:48.210302 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:48.210267 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wz5w8_b0d2478d-1641-4f16-8abb-c044269fc79b/manager/0.log" Nov 02 22:30:48.991816 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:48.991781 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb_53fe7526-a649-4f45-8b6f-d1fb885ca908/istio-proxy/0.log" Nov 02 22:30:49.793230 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:49.793186 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:49.943761 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:49.943732 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:49.991845 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:49.991821 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-27kg9_f47c0164-57f8-4885-a5df-30433184dc46/istio-proxy/0.log" Nov 02 22:30:50.693310 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:50.693275 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hft2d_39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3/istio-proxy/0.log" Nov 02 22:30:50.920183 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:50.920151 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-js4dr_be8ea2f8-52aa-4e10-bca6-b01563cd940f/istio-proxy/0.log" Nov 02 22:30:52.103106 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:52.103074 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zt7w9_b918a37d-859a-4630-a865-e92121ba6073/istio-proxy/0.log" Nov 02 22:30:52.328227 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:52.328197 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth_edd2ef9b-7d22-468c-8eeb-bf7161bbab07/istio-proxy/0.log" Nov 02 22:30:53.849264 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:53.849233 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-tng69_30f6edef-f667-44fa-8cea-f87ba914b7b8/main/0.log" Nov 02 22:30:55.420093 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:55.420063 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh_1ca7a877-0358-45c1-949e-b32728684968/main/0.log" Nov 02 22:30:56.310981 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:30:56.310937 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c57f8d5-mq6lj_33243403-0984-43ce-90f3-04befaf56f63/authorino/0.log" Nov 02 22:31:05.236956 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:05.236921 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bc68ffc7c-pmmp6_636400c6-9f4c-476d-964b-2bf5a279c586/manager/0.log" Nov 02 22:31:06.449429 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:06.449398 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9f8dffcbc-95qc8_f0776fe1-d7f2-4f43-b0e0-a9495263c354/manager/0.log" Nov 02 22:31:06.704675 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:06.704598 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:08.506659 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:08.506628 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-tsmw6_e78e8c53-0934-47ee-9565-c50c1912757d/revoke-keys/0.log" Nov 02 22:31:08.728211 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:08.728145 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-frg57_5149b207-d098-403e-8c57-ed43a9d27406/revoke-keys/0.log" Nov 02 22:31:08.950468 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:08.950437 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dr87f_ecf8ce91-f038-4559-969c-4e29fa65004e/revoke-keys/0.log" Nov 02 22:31:10.280159 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:10.280130 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c57f8d5-mq6lj_33243403-0984-43ce-90f3-04befaf56f63/authorino/0.log" Nov 02 22:31:10.613386 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:31:10.613348 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:31:10.744081 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:10.744051 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-g8lw8_73d816d8-936a-4ff8-a6e6-f49cb87788a5/manager/0.log" Nov 02 22:31:10.955404 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:10.955340 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-4zzv5_2bff1a62-9aa4-478c-a9a9-f95e1c1fd094/manager/0.log" Nov 02 22:31:11.164933 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:11.164906 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6d9v2_e000b5aa-e884-4263-8bc7-1b64c302bd10/kuadrant-console-plugin/0.log" Nov 02 22:31:11.388061 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:11.388025 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5qm9c_6298a9da-6f99-4b43-93d0-7082cd03311b/manager/0.log" Nov 02 22:31:11.870010 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:11.869959 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wz5w8_b0d2478d-1641-4f16-8abb-c044269fc79b/manager/0.log" Nov 02 22:31:12.439157 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:12.439124 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb_53fe7526-a649-4f45-8b6f-d1fb885ca908/istio-proxy/0.log" Nov 02 22:31:13.426277 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:13.426246 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-27kg9_f47c0164-57f8-4885-a5df-30433184dc46/istio-proxy/0.log" Nov 02 22:31:14.131824 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:14.131785 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hft2d_39be6ac1-bfaf-40e3-a1cc-cbb158c2b3f3/istio-proxy/0.log" Nov 02 22:31:14.395153 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:14.395063 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-js4dr_be8ea2f8-52aa-4e10-bca6-b01563cd940f/istio-proxy/0.log" Nov 02 22:31:15.552939 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:15.552907 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zt7w9_b918a37d-859a-4630-a865-e92121ba6073/istio-proxy/0.log" Nov 02 22:31:15.775536 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:15.775499 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-j8dth_edd2ef9b-7d22-468c-8eeb-bf7161bbab07/istio-proxy/0.log" Nov 02 22:31:17.059911 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:17.059880 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-tng69_30f6edef-f667-44fa-8cea-f87ba914b7b8/main/0.log" Nov 02 22:31:18.660189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:18.660156 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-m8phh_1ca7a877-0358-45c1-949e-b32728684968/main/0.log" Nov 02 22:31:40.614261 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:31:40.614219 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:31:45.372811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:45.372778 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9f8dffcbc-95qc8_f0776fe1-d7f2-4f43-b0e0-a9495263c354/manager/0.log" Nov 02 22:31:45.937220 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:45.937190 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9f8dffcbc-95qc8_f0776fe1-d7f2-4f43-b0e0-a9495263c354/manager/0.log" Nov 02 22:31:56.420215 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:31:56.420186 4441 scope.go:122] "RemoveContainer" containerID="a679730c0ff9478081f4ace568584684a99d1d86b2b3f8e79908720aadae5c32" Nov 02 22:32:07.058915 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:07.058882 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:10.614403 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:32:10.614368 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:32:21.398599 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:21.398564 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9f8dffcbc-95qc8_f0776fe1-d7f2-4f43-b0e0-a9495263c354/manager/0.log" Nov 02 22:32:21.957722 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:21.957690 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9f8dffcbc-95qc8_f0776fe1-d7f2-4f43-b0e0-a9495263c354/manager/0.log" Nov 02 22:32:23.215627 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.215582 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nvhb8/must-gather-t6qvd"] Nov 02 22:32:23.216185 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216165 4441 cpu_manager.go:414] "containerMap: removing container" podUID="5149b207-d098-403e-8c57-ed43a9d27406" containerName="revoke-keys" Nov 02 22:32:23.216243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216188 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5149b207-d098-403e-8c57-ed43a9d27406" containerName="revoke-keys" Nov 02 22:32:23.216243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216204 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216209 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216227 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216243 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216232 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216302 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216311 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216318 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="5149b207-d098-403e-8c57-ed43a9d27406" containerName="revoke-keys" Nov 02 22:32:23.216413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216391 4441 cpu_manager.go:414] "containerMap: removing container" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216413 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216396 4441 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.216568 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.216442 4441 memory_manager.go:365] "RemoveStaleState removing state" podUID="0094b188-7f7c-4cf2-81dc-4817a3ae5772" containerName="cleanup" Nov 02 22:32:23.219519 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.219495 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.225811 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.225784 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:23.269176 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.269125 4441 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nvhb8\"/\"default-dockercfg-jq77d\"" Nov 02 22:32:23.316472 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.316445 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nvhb8\"/\"kube-root-ca.crt\"" Nov 02 22:32:23.316601 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.316505 4441 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nvhb8\"/\"openshift-service-ca.crt\"" Nov 02 22:32:23.319551 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.319529 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nvhb8/must-gather-t6qvd"] Nov 02 22:32:23.324985 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.324952 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13afba2c-73e6-4f2b-b6a7-68544de9600c-must-gather-output\") pod \"must-gather-t6qvd\" (UID: \"13afba2c-73e6-4f2b-b6a7-68544de9600c\") " pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.325047 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.325004 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrdb\" (UniqueName: \"kubernetes.io/projected/13afba2c-73e6-4f2b-b6a7-68544de9600c-kube-api-access-twrdb\") pod \"must-gather-t6qvd\" (UID: \"13afba2c-73e6-4f2b-b6a7-68544de9600c\") " pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.351189 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.351163 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:23.425742 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.425719 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13afba2c-73e6-4f2b-b6a7-68544de9600c-must-gather-output\") pod \"must-gather-t6qvd\" (UID: \"13afba2c-73e6-4f2b-b6a7-68544de9600c\") " pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.425853 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.425757 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrdb\" (UniqueName: \"kubernetes.io/projected/13afba2c-73e6-4f2b-b6a7-68544de9600c-kube-api-access-twrdb\") pod \"must-gather-t6qvd\" (UID: \"13afba2c-73e6-4f2b-b6a7-68544de9600c\") " pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.426090 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.426074 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13afba2c-73e6-4f2b-b6a7-68544de9600c-must-gather-output\") pod \"must-gather-t6qvd\" (UID: \"13afba2c-73e6-4f2b-b6a7-68544de9600c\") " pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.480221 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.480162 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrdb\" (UniqueName: \"kubernetes.io/projected/13afba2c-73e6-4f2b-b6a7-68544de9600c-kube-api-access-twrdb\") pod \"must-gather-t6qvd\" (UID: \"13afba2c-73e6-4f2b-b6a7-68544de9600c\") " pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.534203 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.534180 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nvhb8/must-gather-t6qvd" Nov 02 22:32:23.896746 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.896717 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nvhb8/must-gather-t6qvd"] Nov 02 22:32:23.897512 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:32:23.897480 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13afba2c_73e6_4f2b_b6a7_68544de9600c.slice/crio-c84bf06976bf488831e219ac959031916334b07cc8e671a76de050d18924ff3a WatchSource:0}: Error finding container c84bf06976bf488831e219ac959031916334b07cc8e671a76de050d18924ff3a: Status 404 returned error can't find the container with id c84bf06976bf488831e219ac959031916334b07cc8e671a76de050d18924ff3a Nov 02 22:32:23.899460 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:23.899439 4441 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:32:24.780635 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:24.780603 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nvhb8/must-gather-t6qvd" event={"ID":"13afba2c-73e6-4f2b-b6a7-68544de9600c","Type":"ContainerStarted","Data":"c84bf06976bf488831e219ac959031916334b07cc8e671a76de050d18924ff3a"} Nov 02 22:32:25.788107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:25.788068 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nvhb8/must-gather-t6qvd" event={"ID":"13afba2c-73e6-4f2b-b6a7-68544de9600c","Type":"ContainerStarted","Data":"a3f1bc9fbc9ac87834115e4899cd0e921e73d72fe6c8e5e708015682bbe6161d"} Nov 02 22:32:25.788107 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:25.788111 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nvhb8/must-gather-t6qvd" event={"ID":"13afba2c-73e6-4f2b-b6a7-68544de9600c","Type":"ContainerStarted","Data":"503194c712b736bc0945f9302583142da5aa5ab99d1e8364c4b30299d10d0628"} Nov 02 22:32:25.897167 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:25.897117 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nvhb8/must-gather-t6qvd" podStartSLOduration=1.547597546 podStartE2EDuration="2.897103813s" totalImagesPullingTime="1.349506267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:25.894579102 +0000 UTC m=+4775.450207467" watchObservedRunningTime="2026-11-02 22:32:25.897103813 +0000 UTC m=+4775.452732150" Nov 02 22:32:32.514993 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:32.510256 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lsl4s_cbf2c02b-9cee-4a8b-a809-d2b52ec9235b/global-pull-secret-syncer/0.log" Nov 02 22:32:32.766099 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:32.765988 4441 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:34.919045 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:34.917800 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h6889_b97770c2-375e-409d-9795-5cea674e8f2d/konnectivity-agent/0.log" Nov 02 22:32:36.956787 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:36.956742 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-225.us-west-2.compute.internal_1b27bf1c010fcac7e9afcd7fd0096491/haproxy/0.log" Nov 02 22:32:40.650752 ip-10-0-137-225 kubenswrapper[4441]: E1102 22:32:40.650706 4441 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v8947" containerName="extractor" Nov 02 22:32:42.022550 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.022513 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c57f8d5-mq6lj_33243403-0984-43ce-90f3-04befaf56f63/authorino/0.log" Nov 02 22:32:42.345049 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.345009 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-g8lw8_73d816d8-936a-4ff8-a6e6-f49cb87788a5/manager/0.log" Nov 02 22:32:42.428132 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.428098 4441 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh"] Nov 02 22:32:42.433734 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.433708 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.528478 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.528433 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-proc\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.528662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.528580 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dwh\" (UniqueName: \"kubernetes.io/projected/6d54a806-3bc0-4aab-806d-99408f25293d-kube-api-access-n8dwh\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.528662 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.528628 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-lib-modules\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.528762 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.528671 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-podres\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.528812 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.528795 4441 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-sys\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.532250 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.532220 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh"] Nov 02 22:32:42.630146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630070 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dwh\" (UniqueName: \"kubernetes.io/projected/6d54a806-3bc0-4aab-806d-99408f25293d-kube-api-access-n8dwh\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630146 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630112 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-lib-modules\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630148 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-podres\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630189 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-sys\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630252 4441 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-proc\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630266 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-lib-modules\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630270 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-podres\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630305 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-sys\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.630367 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.630348 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d54a806-3bc0-4aab-806d-99408f25293d-proc\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.683983 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.683939 4441 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dwh\" (UniqueName: \"kubernetes.io/projected/6d54a806-3bc0-4aab-806d-99408f25293d-kube-api-access-n8dwh\") pod \"perf-node-gather-daemonset-876zh\" (UID: \"6d54a806-3bc0-4aab-806d-99408f25293d\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.745983 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.745950 4441 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:42.767883 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:42.767849 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-4zzv5_2bff1a62-9aa4-478c-a9a9-f95e1c1fd094/manager/0.log" Nov 02 22:32:43.116808 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:43.116777 4441 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh"] Nov 02 22:32:43.117851 ip-10-0-137-225 kubenswrapper[4441]: W1102 22:32:43.117816 4441 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6d54a806_3bc0_4aab_806d_99408f25293d.slice/crio-1fbd8f54351a0c4531716ec3edbbba4e04d5138a0907a9f7f109dfdf4f6a8930 WatchSource:0}: Error finding container 1fbd8f54351a0c4531716ec3edbbba4e04d5138a0907a9f7f109dfdf4f6a8930: Status 404 returned error can't find the container with id 1fbd8f54351a0c4531716ec3edbbba4e04d5138a0907a9f7f109dfdf4f6a8930 Nov 02 22:32:43.190414 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:43.190392 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6d9v2_e000b5aa-e884-4263-8bc7-1b64c302bd10/kuadrant-console-plugin/0.log" Nov 02 22:32:43.639733 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:43.639702 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5qm9c_6298a9da-6f99-4b43-93d0-7082cd03311b/manager/0.log" Nov 02 22:32:43.947496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:43.947404 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" event={"ID":"6d54a806-3bc0-4aab-806d-99408f25293d","Type":"ContainerStarted","Data":"725631ef98cc093c7693703252a3df53b2488c6e5985a0c3db047988e90363f1"} Nov 02 22:32:43.947496 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:43.947471 4441 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" event={"ID":"6d54a806-3bc0-4aab-806d-99408f25293d","Type":"ContainerStarted","Data":"1fbd8f54351a0c4531716ec3edbbba4e04d5138a0907a9f7f109dfdf4f6a8930"} Nov 02 22:32:43.947685 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:43.947603 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:44.055446 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:44.055394 4441 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" podStartSLOduration=2.055380384 podStartE2EDuration="2.055380384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:44.053691615 +0000 UTC m=+4793.609319952" watchObservedRunningTime="2026-11-02 22:32:44.055380384 +0000 UTC m=+4793.611008720" Nov 02 22:32:44.506311 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:44.506280 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wz5w8_b0d2478d-1641-4f16-8abb-c044269fc79b/manager/0.log" Nov 02 22:32:49.966472 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:49.966440 4441 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-876zh" Nov 02 22:32:51.923089 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:51.923061 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:32:51.930905 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:51.930879 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:32:51.933990 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:51.933954 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac7wzz_b0d1e821-a731-477d-90ff-53a3fd32c8ab/extract/0.log" Nov 02 22:32:51.942896 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:51.942871 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9j4k6_7720b08e-742a-4af9-9dd8-deb90a7d084e/extract/0.log" Nov 02 22:32:51.943033 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:51.942934 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:32:51.960342 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:51.960319 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wgl8r_a8c954c9-200d-4823-b029-54cdebd356aa/ovn-acl-logging/0.log" Nov 02 22:32:55.203102 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:55.203070 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rs2wz_24ae74e6-4a58-4e6e-9a7a-e8b33853846e/node-exporter/0.log" Nov 02 22:32:55.622899 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:55.622867 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rs2wz_24ae74e6-4a58-4e6e-9a7a-e8b33853846e/kube-rbac-proxy/0.log" Nov 02 22:32:56.045118 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:56.045034 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rs2wz_24ae74e6-4a58-4e6e-9a7a-e8b33853846e/init-textfile/0.log" Nov 02 22:32:56.441644 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:32:56.441610 4441 scope.go:122] "RemoveContainer" containerID="545468ed6625bd2129af82269f803d6d0eb8ee836b39d3dc261d9ab524e0aa43" Nov 02 22:33:00.704882 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:33:00.704836 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vb59j_843684e3-80f3-46fe-9441-fc2b5abaf6dd/prometheus-operator/0.log" Nov 02 22:33:01.126557 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:33:01.126528 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vb59j_843684e3-80f3-46fe-9441-fc2b5abaf6dd/kube-rbac-proxy/0.log" Nov 02 22:33:01.550613 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:33:01.550508 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-thzlz_716b4f26-9f07-47ef-837f-11d591f9feea/prometheus-operator-admission-webhook/0.log" Nov 02 22:33:04.945443 ip-10-0-137-225 kubenswrapper[4441]: I1102 22:33:04.945345 4441 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898fs5nlb_53fe7526-a649-4f45-8b6f-d1fb885ca908/istio-proxy/0.log"